Abstract:
With very advanced technology, a very large population of people living happy lives could be sustained in the accessible region of the universe. For every year that development of such technologies and colonization of the universe is delayed, there is therefore a corresponding opportunity cost: a potential good, lives worth living, is not being realized. Given some plausible assumptions, this cost is extremely large. However, the lesson for standard utilitarians is not that we ought to maximize the pace of technological development, but rather that we ought to maximize its safety, i.e. the probability that colonization will eventually occur. This goal has such high utility that standard utilitarians ought to focus all their efforts on it. Utilitarians of a 'personaffecting' stripe should accept a modified version of this conclusion. Some mixed ethical views, which combine utilitarian considerations with other criteria, will also be committed to a similar bottom line.
 I. THE RATE OF LOSS OF POTENTIAL LIVES As I write these words, suns are illuminating and heating empty rooms, unused energy is being flushed down black holes, and our great common endowment of negentropy is Russian, Portuguese, Bosnian Translations:

Astronomical Waste: The Opportunity Cost of Delayed

2 being irreversibly degraded into entropy on a cosmic scale. These are resources that an advanced civilization could have used to create value-structures, such as sentient beings living worthwhile lives. The rate of this loss boggles the mind. One recent paper speculates, using loose theoretical considerations based on the rate of increase of entropy, that the loss of potential human lives in our own galactic supercluster is at least ~10 46 per century of delayed colonization.  1  This estimate assumes that all the lost entropy could have been used for productive purposes, although no currently known technological mechanisms are even remotely capable of doing that. Since the estimate is meant to be a lower bound, this radically unconservative assumption is undesirable. We can, however, get a lower bound more straightforwardly by simply counting the number or stars in our galactic supercluster and multiplying this number with the amount of computing power that the resources of each star could be used to generate using technologies for whose feasibility a strong case has already been made. We can then divide this total with the estimated amount of computing power needed to simulate one human life. As a rough approximation, let us say the Virgo Supercluster contains 10 13 stars. One estimate of the computing power extractable from a star and with an associated planet-sized computational structure, using advanced molecular nanotechnology 2 , is 10 42 operations per second. 3 A typical estimate of the human brain's processing power is roughly 10 17 operations per second or less. 4 Not much more seems to be needed to simulate the relevant parts of the environment in sufficient detail to enable the simulated minds to have experiences indistinguishable from typical current human experiences. 5 3 Given these estimates, it follows that the potential for approximately 10 38 human lives is lost every century that colonization of our local supercluster is delayed; or equivalently, about 10 29 potential human lives per second. While this estimate is conservative in that it assumes only computational mechanisms whose implementation has been at least outlined in the literature, it is useful to have an even more conservative estimate that does not assume a non-biological instantiation of the potential persons. Suppose that about 10 10 biological humans could be sustained around an average star. Then the Virgo Supercluster could contain 10 23 biological humans. This corresponds to a loss of potential of over 10 13 potential human lives per second of delayed colonization. What matters for present purposes is not the exact numbers but the fact that they are huge. Even with the most conservative estimate, assuming a biological implementation of all persons, the potential for over ten trillion potential human beings is lost for every second of postponement of colonization of our supercluster.  6
 II. THE OPPORTUNITY COST OF DELAYED COLONIZATION From a utilitarian perspective, this huge loss of potential human lives constitutes a correspondingly huge loss of potential value. I am assuming here that the human lives that could have been created would have been worthwhile ones. Since it is commonly supposed that even current human lives are typically worthwhile, this is a weak assumption. Any civilization advanced enough to colonize the local supercluster would likely also have the ability to establish at least the minimally favorable conditions required for future lives to be worth living.
 4 The effect on total value, then, seems greater for actions that accelerate technological development than for practically any other possible action. Advancing technology (or its enabling factors, such as economic productivity) even by such a tiny amount that it leads to colonization of the local supercluster just one second earlier than would otherwise have happened amounts to bringing about more than 10 29 human lives (or 10 13 human lives if we use the most conservative lower bound) that would not otherwise have existed. Few other philanthropic causes could hope to match that level of utilitarian payoff. Utilitarians are not the only ones who should strongly oppose astronomical waste. There are many views about what has value that would concur with the assessment that the current rate of wastage constitutes an enormous loss of potential value. For example, we can take a thicker conception of human welfare than commonly supposed by utilitarians (whether of a hedonistic, experientialist, or desire-satisfactionist bent), such as a conception that locates value also in human flourishing, meaningful relationships, noble character, individual expression, aesthetic appreciation, and so forth. So long as the evaluation function is aggregative (does not count one person's welfare for less just because there are many other persons in existence who also enjoy happy lives) and is not relativized to a particular point in time (no time-discounting), the conclusion will hold. These conditions can be relaxed further. Even if the welfare function is not perfectly aggregative (perhaps because one component of the good is diversity, the marginal rate of production of which might decline with increasing population size), it can still yield a similar bottom line provided only that at least some significant component of the good is sufficiently aggregative. Similarly, some degree of timediscounting future goods could be accommodated without changing the conclusion. 7
 III. THE CHIEF GOAL FOR UTILITARIANS SHOULD BE TO REDUCE EXISTENTIAL RISK In light of the above discussion, it may seem as if a utilitarian ought to focus her efforts on accelerating technological development. The payoff from even a very slight success in this endeavor is so enormous that it dwarfs that of almost any other activity. We appear to have a utilitarian argument for the greatest possible urgency of technological development. However, the true lesson is a different one. If what we are concerned with is (something like) maximizing the expected number of worthwhile lives that we will create, then in addition to the opportunity cost of delayed colonization, we have to take into account the risk of failure to colonize at all. We might fall victim to an existential risk, one where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential. 8 Because the lifespan of galaxies is measured in billions of years, whereas the time-scale of any delays that we could realistically affect would rather be measured in years or decades, the consideration of risk trumps the consideration of opportunity cost. For example, a single percentage point of reduction of existential risks would be worth (from a utilitarian expected utility point-ofview) a delay of over 10 million years. Therefore, if our actions have even the slightest effect on the probability of eventual colonization, this will outweigh their effect on when colonization takes place.
 6 For standard utilitarians, priority number one, two, three and four should consequently be to reduce existential risk. The utilitarian imperative 'Maximize expected aggregate utility!' can be simplified to the maxim 'Minimize existential risk!'.
 IV. IMPLICATIONS FOR AGGREGATIVE PERSON-AFFECTING VIEWS The argument above presupposes that our concern is to maximize the total amount of well-being. Suppose instead that we adopt a 'person-affecting' version of utilitarianism, according to which our obligations are primarily towards currently existing persons and to those persons that will come to exist. 9 On such a person-affecting view, human extinction would be bad only because it makes past or ongoing lives worse, not because it constitutes a loss of potential worthwhile lives. What ought someone who embraces this doctrine do? Should he emphasize speed or safety, or something else? To answer this, we need to consider some further matters. Suppose one thinks that the probability is negligible that any existing person will survive long enough to get to use a significant portion of the accessible astronomical resources, which, as described in opening section of this paper, are gradually going to waste. Then one's reason for minimizing existential risk is that sudden extinction would off cut an average of, say, 40 years from each of the current (six billion or so) human lives. 10 While this would certainly be a large disaster, it is in the same big ballpark as other ongoing human tragedies, such as world poverty, hunger and disease. On this assumption, then, a personaffecting utilitarian should regard reducing existential risk as a very important but not completely dominating concern. There would in this case be no easy answer to what he ought to do. Where he ought to focus his efforts would depend on detailed calculations about which area of philanthropic activity he would happen to be best placed to make a contribution to. Arguably, however, we ought to assign a non-negligible probability to some current people surviving long enough to reap the benefits of a cosmic diaspora. A socalled technological 'singularity' might occur in our natural lifetime 11 , or there could be a breakthrough in life-extension, brought about, perhaps, as result of machine-phase nanotechnology that would give us unprecedented control over the biochemical processes in our bodies and enable us to halt and reverse the aging process. 12 Many leading technologists and futurist thinkers give a fairly high probability to these developments happening within the next several decades.  13  Even if you are skeptical about their prognostications, you should consider the poor track record of technological forecasting. In view of the well-established unreliability of many such forecasts, it would seem unwarranted to be so confident in one's prediction that the requisite breakthroughs will not occur in our time as to give the hypothesis that they will a probability of less than, say, 1%. The expected utility of a 1% chance of realizing an astronomically large good could still be astronomical. But just how good would it be for (some substantial subset of) currently living people to get access to astronomical amounts of resources? The answer is not obvious. On the one hand, one might reflect that in today's world, the marginal utility for an individual of material resources declines quite rapidly ones his basic needs have been met. Bill Gate's level of well-being does not seem to dramatically exceed that of many a person of much more modest means. On the other hand, advanced technologies of the sorts that would most likely be deployed by the time we could colonize the local supercluster may well provide new ways of converting resources into well-being. In particular, material resources could be used to greatly expand our mental capacities and to indefinitely prolong our subjective lifespan. And it is by no means clear that the marginal utility of extended healthspan and increased mental powers must be sharply declining above some level. If there is no such decline in marginal utility, we have to conclude that the expected utility to current individuals 14 of successful colonization of our supercluster is astronomically great, and this conclusion holds even if one gives a fairly low probability to that outcome. A long shot it may be, but for an expected utility maximizer, the benefit of living for perhaps billions of subjective years with greatly expanded capacities under fantastically favorable conditions could more than make up for the remote prospects of success. Now, if these assumptions are made, what follows about how a person-affecting utilitarian should act? Clearly, avoiding existential calamities is important, not just because it would truncate the natural lifespan of six billion or so people, but also -and given the assumptions this is an even weightier consideration -because it would extinguish the chance that current people have of reaping the enormous benefits of eventual colonization. However, by contrast to the total utilitarian, the person-affecting utilitarian would have to balance this goal with another equally important desideratum, namely that of maximizing the chances of current people surviving to benefit from the colonization. For the person-affecting utilitarian, it is not enough that humankind survives to colonize; it is crucial that extant people be saved. This should lead her to emphasize speed of technological development, since the rapid arrival of advanced technology would surely be needed to help current people stay alive until the fruits of 9 colonization could be harvested. If the goal of speed conflicts with the goal of global safety, the total utilitarian should always opt to maximize safety, but the person-affecting utilitarian would have to balance the risk of people dying of old age with the risk of them succumbing in a species-destroying catastrophe. Mixed ethical views, which also incorporate non-utilitarian elements, might or might not yield one of these bottom lines depending on the nature of what is added. 15 1 M. Cirkovic, 'Cosmological Forecast and its Practical Significance', Journal of  Evolution and Technology, xii (2002) , http://www.jetpress.org/volume12/CosmologicalForecast.pdf. 10 7 Utilitarians commonly regard time-discounting as inappropriate in evaluating moral goods (see e.g. R. B.  Brandt, Morality, Utilitarianism, and Rights, Cambridge, 1992, pp. 23f.) . However, it is not clear that utilitarians can avoid compromising on this principle in view of the possibility that our actions could conceivably have consequences for an infinite number of persons (a possibility that we set aside for the purposes of this paper).  8 N. Bostrom, 'Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards', Journal of
			 K. E. Drexler, Nanosystems: Molecular Machinery, Manufacturing, and Computation, New York, John Wiley & Sons, Inc., 1992. 3 R. J. Bradbury, 'Matrioshka Brains', Manuscript, 2002, http://www.aeiveos.com/~bradbury/MatrioshkaBrains/MatrioshkaBrains.html 4 N. Bostrom, 'How Long Before Superintelligence?', International Journal of Futures Studies ii (1998); R. Kurzweil, The Age of Spiritual Machines: When Computers Exceed Human Intelligence, New York, Viking, 1999. The lower estimate is in H. Moravec, Robot: Mere Machine to Transcendent Mind, Oxford, 1999. 5 N. Bostrom, 'Are You Living in a Simulation?', Philosophical Quarterly, liii (211). See also http://www.simulation-argument.com.6  The Virgo Supercluster contains only a small part of the colonizable resources in the universe, but it is sufficiently big to make the point. The bigger the region we consider, the less certain we can be that significant parts of it will not have been colonized by a civilization of non-terrestrial origin by the time we could get there.
			 I'm grateful for comments from John Broome, Milan Cirkovic, Roger Crisp, Robin Hanson, and James Lenman, and for the financial support of a British Academy Postdoctoral Award.


Abstract: It is incredibly easy for a system designer to misspecify the objective for an autonomous system ("robot"), thus motivating the desire to have the robot learn the objective from human behavior instead. Recent work has suggested that people have an interest in the robot performing well, and will thus behave pedagogically, choosing actions that are informative to the robot. In turn, robots benefit from interpreting the behavior by accounting for this pedagogy. In this work, we focus on misspecification: we argue that robots might not know whether people are being pedagogic or literal and that it is important to ask which assumption is safer to make. We cast objective learning into the more general form of a common-payoff game between the robot and human, and prove that in any such game literal interpretation is more robust to misspecification. Experiments with human data support our theoretical results and point to the sensitivity of the pedagogic assumption.

Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learning

INTRODUCTION It is notoriously difficult for system designers to directly specify the correct objective for a system  (Krakovna, 2018; Lehman et al., 2018; Clark & Amodei, 2017) . This difficulty has sparked a line of work that instead aims to infer the correct objective from other forms of human input, such as demonstrations  (Ng & Russell, 2000; Abbeel & Ng, 2004; Ziebart et al., 2008) , comparisons  (Wirth et al., 2017; Sadigh et al., 2017; Christiano et al., 2017) , or corrections  (Bajcsy et al., 2017; Jain et al., 2015) . These methods operate under the assumption that human behavior is near-optimal with respect to the objective to be inferred. This assumption makes sense in many domains, like an autonomous car learning an objective function for driving through observing human drivers  (Levine & Koltun, 2012) . However, recent work shows that this assumption may not hold in collaborative settings  (Ho et al., 2016) , in which the human is aware that the robot needs to learn. In such settings, the person might optimize for teaching the robot about the objective, which is not the same as directly optimizing the objective itself  (Dragan et al., 2013b; Hadfield-Menell et al., 2016; Ho et al., 2016) . Figure  1  shows an example of the difference between the two. When the human optimizes for the objective, she takes any optimal path to the goal. When the human optimizes for teaching the objective, she takes the path that best signals the objective. We refer to these two types of human behavior as literal and pedagogic, and note that the robot can interpret human behavior using either model: 1. The literal human directly optimizes for the objective. 2. The literal robot infers the objective while assuming the human is literal. 3. The pedagogic human optimizes for teaching the literal robot the objective. 4. The pedagogic robot (sometimes called "pragmatic" ) infers the objective while assuming the human is pedagogic. . . . In general, this recursion could go on further, and in theory, it is always better for the human and robot to be at a deeper level of recursion. The potential for increased performance suggests that we should try to make our robots more pedagogic. Indeed, this is the direction suggested and pursued by ;  In this scenario, the robot infers whether it is ok to walk on the grass or not from a demonstration provided by the human (it already knows pavement is always ok). (a) The literal and pedagogic human's demonstrations. When grass is ok, the literal human is equally likely to walk on the grass or pavement. On the other hand, when grass is ok, the pedagogic human always walks on the grass in order to signal that grass is ok. (b) The beliefs of the literal robot LR and pedagogic robot PR after observing a demonstration. LR and PR assume human is literal and pedagogic, respectively. LR and PR differ in how strong their beliefs are after witnessing the human walk on pavement, leading to different problems when the human is misspecified. If the human is literal, but the robot is pedagogic, then it makes too strong of an inference. On the other hand, if the human is pedagogic, but the robot is literal, then it makes too weak of an inference. However, the increased performance is contingent on the human being pedagogic. In practice, it may be difficult to know whether the human is acting literally or pedagogically, and furthermore, different humans may behave differently. We argue that the robot should be robust to misspecification of the human, and thus it is important to ask which assumption -literal or pedagogic -is safer to make. Which is worse-interpreting a pedagogic human literally (literal robot + pedagogic human) or interpreting a literal human pedagogically (pedagogic robot + literal human)? In both cases, the model of the human is incorrect, and we might expect that which is better depends on the context and the task; surprisingly, we are able to prove that regardless of the task, the literal robot is more robust, suggesting that it may be safer to simply use a literal robot. Our contributions are the following: • Section 2: Theoretical Analysis. We cast objective learning into the more general form of a commonpayoff game between the human and robot. We then prove that a pedagogic robot and a literal human always do worse than a literal robot and a pedagogic human, showing that misspecification is worse in one direction than the other. • Section 3: Empirical Analysis. We test the effects of misspecification on data from human teaching. The data confirms that assuming pedagogic behavior when people are literal is worse than assuming literal behavior when people are pedagogic. Surprisingly, we find that the literal robot does better than the pedagogic robot even when people are trying to be pedagogic, because of the discrepancy be-tween the pedagogic model of humans and real human behavior. • Section 4: Theoretical vs Empirical. Our empirical results are surprising, in that, the pedagogic model is a state of the art cognitive science model that is fit to the human data and has relatively high predictive accuracy, yet the pedagogic robot does worse than the literal robot with humans who are trying to be pedagogic. We use our theory to derive a hypothesis for why this could be. The hypothesis is that in practice different humans vary in how literal or pedagogic they are, which our theory implies will degrade the performance of the pedagogic robot more than the literal robot. We find positive evidence for this hypothesis, indicating that robustness to a population of humans is an important consideration in choosing a pedagogic versus literal robot. • Section 5: Can we "fix" the pedagogic robot? An intuitive idea for improving the pedagogic robot is to give it a model of the pedagogic human that has higher predictive accuracy. For example, what if instead of assuming all people are pedagogic, the robot estimated how pedagogic each person is? Unfortunately, we find that this makes no difference. And in fact, we show that better models can actually worsen performance due to a subtle, yet remarkable fact: a human model with higher predictive accuracy does not necessarily imply higher inferential accuracy for the robot. In conclusion, we found that not only are pedagogic robots less robust to the human's recursion level, they can also perform worse when people are actually being pedagogic -even with a state of the art pedagogic model tuned to human data. This points to a surprising brittleness of the pedagogic assumption. Further, we point out that a more predictive human model will not necessarily solve the problem because improving the model's predictive accuracy does not necessarily lead to better robot inference. The difficulty of objective specification was an important motivation for pursuing objective learning in the first place. But in our pursuit of objective learning, we ought to be careful to not simply trade the problem of objective specification for the equally, if not more, difficult problem of human specification. In practice, humans will deviate from our models of them, and thus, it is important to understand which assumptions are robust and which are not. Rather than trying to make the robot more pedagogic, which requires brittle assumptions on the human, an alternative direction may be to help the human become more pedagogic. How can we make robots that are easier for humans to teach? Perhaps simpler robots are actually better, if it means that humans can more easily teach them.
 THEORETICAL ANALYSIS In this section, we cast objective learning into the more general form of a common-payoff 1 game between the robot and human. We then formalize the literal/pedagogic robot/human, and prove that in any common-payoff game a literal robot and pedagogic human perform better than a pedagogic robot and literal human.
 GENERALIZING OBJECTIVE LEARNING TO CI(RL) Before proceeding to our proof, we give background on how common-payoff games generalize standard objective learning. In particular, objective learning can be modeled as a cooperative inverse reinforcement learning (CIRL) game  (Hadfield-Menell et al., 2016) , a commonpayoff game in which only the human knows an objective/reward function r. Formally, a CIRL game is defined as a tuple S, {A H , A R }, T, {R, r}, P 0 , γ} where S is the set of states, A H and A R are the set of actions available to the human and robot, T (s | s, a H , a R ) is the transition distribution specifying the probability of transitioning to a new state s given the previous state s and the actions a H and a R of both agents, R is the space of reward functions, r : S × A H × A R → R is the shared reward function (known only to H), P 0 is the initial distribution over states and reward functions, and γ is the discount factor. The joint payoff in a CIRL game is traditionally value, expected sum of rewards. Since only the human in CIRL knows the shared reward function, this indirectly incentivizes the human to act in ways that signal the reward function and incentivizes the robot to learn about the reward function from the human's behavior 2 . However, we can also consider a version that directly incentivizes reward inference by making the payoff the accuracy of the robot's inference. To illustrate how objective learning settings are special cases of CIRL games, we give an example for the learning from demonstrations setting. Example 2.1 (Demonstration-CI(RL)). Learning from demonstrations can be modeled as a CIRL game with two phases. In the first phase, the human provides demonstrations. In the second phase... The second formulation (b) is a cooperative inference (CI) problem  (Yang et al., 2018; Wang et al., 2019) . In CI, there is a teacher (e.g human) who is teaching a learner (e.g. robot) a hypothesis r (e.g the reward) via data d (e.g. demonstrations). The human has a distribution over demonstrations p H (d | r) and the robot has a distribution over rewards p R (r | d). Given a starting distribution of human demonstrations, p H 0 (d | r), the optimal solution for these two distributions can be found via fixed-point iteration of the following recursive equations 3 : p R k (r | d) ∝ p H k (d | r) , (1) p H k+1 (d | r) ∝ p R k (r | d) . (2)
 PROOF: LITERAL ROBOTS ARE MORE ROBUST We now proceed to formalize what we mean by literal and pedagogic and to prove that the literal robot is more robust to misspecification of whether the human is literal or pedagogic. Suppose that the human and robot are acting in a cooperative game. Let H and R be the space of policies for the human and robot, respectively. The joint payoff for the human and robot is denoted by U : H × R → R. The joint payoff function could be value, as assumed by CIRL, or accuracy of inference, as assumed by CI. To define literal and pedagogic, we define a set of recursive policies for the human and robot. Let H 0 be a starting human policy. For k ≥ 0, define the recursive policies R k = BR(H k ) , (3) H k+1 = IR(R k ) . (4) We assume that at each level of recursion the robot does a best response BR : H → R. However, for the human, we only assume that at each step she improves over her previous policy. This allows for arbitrary irrationalities, so long as the next policy is at least as good as the previous one. We call this an "improving" response IR : R → H, and define both types of responses below. Definition 2.1. A best response for the robot is a function BR : H → R such that for any human policy H ∈ H and robot policy R ∈ R, U (H, BR(H)) ≥ U (H, R) . Definition 2.2. An improving response for the human is a function IR : R → H such that ∀ k ≥ 0, U (IR(R k ), R k ) ≥ U (H k , R k ) . We give special emphasis to what we call the literal human and robot, H 0 and R 0 , and the pedagogic human and robot, H 1 and R 1 . 4 We now provide an example of the literal and pedagogic policies for the Demonstration-CI setting (Example 2.1b). In this case, the human and robot policies can be modeled by the recursive CI equations (1) and (2). p H 0 (d | r) that she gives a demonstration d is exponentially proportional to the reward of the demonstration, denoted by r(d): p H 0 (d | r) ∝ exp(r(d)) . 2. The literal robot R 0 does a BR to the literal human, i.e. does Bayesian inference assuming the human is literal, p R 0 (r | d) ∝ p H 0 (d | r) , and then uses the posterior mode as its estimate for the reward r. 3. The pedagogic human H 1 picks demonstrations that are informative to the literal robot 5 : p H 1 (d | r) ∝ p R 0 (r | d) . Note this is not a best response, which would unrealistically require the human to choose arg max d p R 0 (r | d). 4. The pedagogic robot R 1 does a BR to the pedagogic human, i.e. does Bayesian inference assuming the human is literal, p R 1 (r | d) ∝ p H 1 (d | r) , and then uses the posterior mode as its estimate for the reward r. We show that for any common-payoff game the payoffs for the literal/pedagogic human/robot pairs have the following ranking: Pedagogic R1, H1 ≥ Literal R0, Pedagogic H1 ≥ Literal R0, H0 ≥ Pedagogic R1, Literal H0 . (5) In particular, a pedagogic robot R 1 and a literal human H 0 always do worse than a literal robot R 0 and a pedagogic human H 1 , showing misspecification is worse one way than the other. The ranking has the following straight-forward proof. Claim 2.1. In any common-payoff game, the ranking of payoffs between a literal/pedagogic human/robot is U (H 1 , R 1 ) ≥ U (H 1 , R 0 ) ≥ U (H 0 , R 0 ) ≥ U (H 0 , R 1 ) . Proof. Since R 1 = BR(H 1 ), we have U (H 1 , R 1 ) ≥ U (H 1 , R 0 ). Since H 1 = IR(R 0 ), we have U (H 1 , R 0 ) ≥ U (H 0 , R 0 ). Since R 0 = BR(H 0 ), we have U (H 0 , R 0 ) ≥ U (H 0 , R 1 ). 5 Typically, the human is modeled as being exponentially more informative: p H 1 (d | r) ∝ exp(p R 0 (r | d)) . This is the form we will use in our experiments.
 EMPIRICAL ANALYSIS In this section, we test our results in practice using experimental data from actual humans, who provide demonstrations that the robot uses to infer the objective from. These experiments are a way of stress-testing our theoretical results, which assumed that at each level of recursion, the robot computes a best response and that the human never does worse than her previous level. In practice, there will be a difference between what humans actually do, and the model of the human, which could cause both the human and robot to break our theoretical assumptions.
 EXPERIMENTAL DESIGN We test the performance of literal/pedagogic robot/human pairs on data from Experiment 2 in (Ho et al., 2016), which was subsequently followed up on in  (Ho et al., 2018) . In the experiment, humans are asked to act in different types of gridworlds. The gridworld has one goal state that is worth 10 points and three other types of tiles (orange, purple, cyan). Each type of tile can each be either "safe" (0 points) or "dangerous" (-2 points). Thus, there are 2 3 = 8 possible reward functions, which are depicted in Figure  2b . Sixty particpants were recruited from Mechanical Turk. The participants are told that they will get two cents of bonus for each point they get. Participants were split into two conditions, a literal and pedagogic condition, depicted in Figure  2a . In the literal condition, the participant only gets points for their own actions in the gridworld. In the pedagogic condition, the participant is told that their demonstration will be shown to another person, a learner, who will then apply what they learn from the demonstration to act in a separate gridworld. The partic-ipant still gets points based on their own actions, but is also told that the number of points the learner receives will be added as a bonus.
 HUMAN AND ROBOT MODELS We model the robot and human following Ho et al.  (2018) . We use the same model parameters that Ho et al. (2018) found to be the best qualitative match to the human demonstrations. Notation. Let S be the set of states and A be the set of actions. The gridworld has a reward function r : S × A × S → R. The optimal Q-value function for a reward function r is denoted by Q * r : S × A → R. The robot has a uniform prior belief over the reward function, i.e, it puts uniform probability on all 2 3 = 8 reward functions depicted in Figure  2b . The human and robot models are as follows. 1. Literal H. At each time step t, the probability H L (a t |s t , r) that the literal human takes action a t given state s t and the reward r is exponentially proportional to the optimal Q-value, H L (a t | s t , r) ∝ exp(Q * r (s t , a t )/τ L ) . (6) The temperature parameter τ L controls how noisy the human is. 2. Literal R. The robot does Bayesian inference, while assuming that the human is literal. The robot's posterior belief at time t + 1 is R t+1 L (r) ∝ H L (a t | s t , r) • T (s t+1 | s t , a t ) • R t L (r) . (7) 3. Pedagogic H. The pedagogic human optimizes a reward function r that trades-off between optimizing the reward in the gridworld and teaching the literal robot the reward. At time step t, the reward is r (s t , a t , s t+1 ) = r(s t , a t , s t+1 ) + κ(R t+1 L (r) − R t L (r)). The parameter κ ≥ 0 controls how "pedagogic" the human is. The probability H P (a t | s t , r) that the pedagogic human takes action a t given state s t and the reward r is exponentially proportional to the optimal Q-value associated with the modified reward r , H P (a t | s t , r) ∝ exp(Q * r (s t , a t )/τ P ) . (8) The temperature parameter τ P controls how noisy the human is. 4. Pedagogic R. The robot does Bayesian inference, while assuming that the human is literal. The robot's posterior belief at time t + 1 is R t+1 P (r) ∝ H P (a t | s t , r) • T (s t+1 | s t , a t ) • R t P (r) . (9)
 RESULTS We evaluate each literal/pedagogic robot/human pair on the accuracy of the robot's inference, i.e, P(r = r), where r is the true reward and r is the robot's guess. We take the robot's guess r to be the mode of its belief, as given by the robot models (  7 ) and (  9 ). We test each pair with both the demonstrations generated by actual humans and demonstrations generated by simulating humans according to the human models (  6 ) and (  7 ). Figure  3  depicts our experimental results 6 . We refer to the actual human as AH, the human model as H, and the robot as R. Consistent with the theory, the performance of pedagogic R and literal AH is (significantly) worse than that of literal R and pedagogic AH, validating that misspecification is worse one way than the other. However, the overall ranking of robot/human pairs does not match the theoretical ranking (Equation  5 ) we expected. Surprisingly, even when AH is pedagogic, pedagogic R performs (insignificantly) worse than literal R. The empirical ranking is Literal R, Pedagogic AH ≥ Pedagogic R, AH ≥ Literal R, AH ≥ Pedagogic R, Literal AH . The empirical ranking carries a stronger implication than our theoretical ranking-it implies that regardless of whether the human is literal or pedagogic, the robot should be literal.  Pedagogic R Pedagogic AH Literal R Pedagogic AH Literal R Literal AH Pedagogic R Literal AH Robot/
 THEORETICAL VS EMPIRICAL In our empirical analysis, we found a perplexing result: the literal robot does better than the pedagogic robot even when people are trying to be pedagogic. How could this be? Figure  4  shows the accuracy of robot/human pairs when the human demonstrations are simulated from the literal H and pedagogic H models described in equations (  6 ) and (  8 ). In the simulations, we find, as expected, that pedagogic R and pedagogic H do better than literal R and pedagogic H. So clearly, the reason pedagogic R does worse in the human experiments is that pedagogic AH (actual humans) is not the same as the pedagogic H (the human model). But, in what way does pedagogic AH deviate from pedagogic H? Why is literal R more robust to the deviation than pedagogic R? We derive a hypothesis from our theory. In the theoretical ranking (Equation  5 ), the performance of literal R and literal/pedagogic H is sandwiched between the performance of pedagogic R + pedagogic H and pedagogic R + literal H. Thus, literal R is more robust to whether H is literal or pedagogic than pedagogic R. This suggests an explanation for why literal R does better, namely, that pedagogic AH is actually sometimes literal! Hypothesis 4.1. Pedagogic AH is actually "in between" literal H and pedagogic H. If this is true, then our theory implies that literal R will be affected less than pedagogic R, potentially explaining why pedagogic R does worse with pedagogic AH than literal R. Pedagogic R Pedagogic H Literal R Pedagogic H Literal R Literal H Pedagogic R Literal H Robot/ To test Hypothesis 4.1, we create two models that are mixtures of the literal and pedagogic model. We then measure how much better the mixture models are at predicting pedagogic AH, and how much the literal and pedagogic R are affected by demonstrations generated from the mixture models.
 DEMONSTRATION MIXTURE MODEL First, we test what we call the "demonstration mixture model". It is possible that the humans in the pedagogic condition from Ho et al. (  2016 ) followed different strategies; some may have attempted to be pedagogic, but others may have simply been literal. If we look at which model, literal or pedagogic, is a better fit on a perindividual basis, we find that 89.7% of literal AH are better described by literal H, but in comparison, only 70.0% of pedagogic AH are better described by pedagogic H. If we simulate pedagogic humans as only following the model 70.0% of the time, then the accuracy of the pedagogic robot drops to 90% (Figure  5a ). The literal robot is hardly impacted.
 ACTION MIXTURE MODEL We now test a more continuous version of the previous setting. We now model humans as acting according to a mixture policy H m between the literal H L and pedagogic H P policies. In particular, at each time t the probability the human picks action a t from state s in an environment with reward r is H M (a t | s t , r) = αH P (a t | s t , r) + (1 − α)H L (a t | s t , r) . ( 10 ) The parameter α is the probability of picking an action according to the pedagogic model. We plot the likelihood of the actual pedagogic human demonstrations as a function of α, (10). The best value of α over the whole population was α P = 0.5 (Figure  6a ). But surprisingly, even a mixture with α = 0.01 or α = 0.99 is far better than either the literal (α = 0) or pedagogic (α = 1) model. In addition, the best estimates for α on an individual basis are somewhat bimodal (Figure  6b ), indicating that there is high individual variation. Figure  5b  shows the performamce of the literal and pedagogic robot as α varies. At the best population level α P = 0.5, we find that the pedagogic robot gets 97% accuracy. However, if we simulate the pedagogic humans using the values of α estimated at an individual level, then the pedagogic robot's accuracy drops to 90%, highlighting the importance of individual variation. The literal robot again remains unaffected.
 DISCUSSION In both the demonstration and action mixture models, we found that a mixture between pedagogic H and literal H was a much better fit to AH. In both cases, when pedagogic H is simulated according to the more accurate mixture model pedagogic R's accuracy drops ten percentage points, but literal R's accuracy hardly changes. Thus our results provide positive evidence for Hypothesis 4.1. However, Hypothesis 4.1 does not explain the full story. The hypothesis can only account for a ten percentage drop in accuracy, but even with this drop, pedagogic R would still be better than literal R. Furthermore, with a cursory glance at Figure  5 , one might be tempted to consider pedagogic R quite robust, as it remains highperforming for large ranges of α. However, as usual, the real problem is the unknown unknowns. Our empirical results imply that there are other ways that humans deviate from the model and that literal R is more robust than pedagogic R to these unknown deviations. Rather than robustness to α, the more compelling reason for choosing to use literal R is robustness to these unknown deviations.
 CAN WE "FIX" THE PEDAGOGIC ROBOT? An intuitive idea for improving the performance of the pedagogic robot R is to give it a model of the pedagogic human H that is more predictive. Unfortunately, it is not so simple. For example, in Section 4.2, we showed that a mixture between the literal and pedagogic human model was a much better fit to actual pedagogic humans than either the literal or pedagogic human model. What if we had pedagogic R use the more accurate mixture model as its model of the pedagogic human? Unfortunately, as shown in Figure  5a  (purple line), even if pedagogic R uses a better predictive model, i.e. the action mixture model, it does not perform any better. In fact, it is possible for pedagogic R to do worse when given a more predictive model of the human. The reason is that a model that is better for predicting behavior (e.g. human demonstrations) is not necessarily better for inferring underlying, latent variables (e.g. the reward function), which is what is important for the robot. This may help explain why pedagogic R does worse than literal R with pedagogic AH, even though the pedagogic H model assumed by pedagogic R is more predictive of pedagogic AH than the literal H model assumed by literal R. To illustrate, suppose there is a latent variable θ ∈ Θ with prior distribution p(θ) and observed data x ∈ X generated by some distribution p(x | θ). In our setting, θ corresponds to the objective and x corresponds to the human input. For simplicity, we assume Θ and X are finite. We have access to a training dataset D = {(θ i , x i )} n i=1 of size n. A predictive model m(x | θ) models the conditional probability of the data x given latent variable θ for all x ∈ X , θ ∈ Θ. In our case, the predictive model is the model of the human. The predictive likelihood L X of a predictive model m is simply the likelihood of the data under the model: L X (m) = n i=1 m(x i | θ i ) . (11) The inferential likelihood is the likelihood of the latent variables after applying Bayes' rule: L Θ (m) = n i=1 m(x i | θ i )p(θ i ) θ m(x i | θ)p(θ) . ( 12 ) Next, we show higher predictive likelihood does not necessarily imply higher inferential likelihood. Claim 5.1 (Predictive vs inferential likelihood). There exist settings in which there are two predictive models m 1 , m 2 such that L X (m 1 ) > L X (m 2 ), but L Θ (m 1 ) < L Θ (m 2 ). Proof. Suppose that Θ = {θ 1 , θ 2 } and X = {x 1 , x 2 , x 3 }, the prior p(θ) is uniform over Θ, and the dataset D contains the following n = 9 items: D = {(θ 1 , x 1 ), (θ 1 , x 1 ), (θ 1 , x 2 ), (θ 2 , x 2 ), (θ 2 , x 2 ), (θ 2 , x 3 ), (θ 2 , x 3 ), (θ 2 , x 3 ), (θ 2 , x 3 )} . Define the models m 1 (x | θ) and m 2 (x | θ) by the following conditional probabilities tables. m 1 (x | θ) x 1 x 2 x 3 θ 1 2/3 1/3 0 θ 2 0 1/3 2/3 m 2 (x | θ) x 1 x 2 x 3 θ 1 2/3 1/3 0 θ 2 0 2/3 1/3 The model m 1 has predictive likelihood L X (m 1 ) = (2/3) 6 (1/3) 3 and inferential likelihood L Θ (m 1 ) = (1/2) 3 . The model m 2 has predictive likelihood L X (m 2 ) = (2/3) 4 (1/3) 5 and inferential likelihood L Θ (m 2 ) = (1/3)(2/3) 3 . Thus, L X (m 1 ) > L X (m 2 ), but L Θ (m 1 ) < L Θ (m 2 ). In our context, Claim 5.1 means that a human model that is better in terms of prediction may actually be worse for the robot to use for inference. An alternative approach could be to directly fit models that optimize for inferential likelihood. Unfortunately, this optimization becomes much trickier because of the normalization over the latent variable space Θ in the denominator of (  12 ), see e.g.  (Dragan et al., 2013a) . Figure1: In this scenario, the robot infers whether it is ok to walk on the grass or not from a demonstration provided by the human (it already knows pavement is always ok). (a) The literal and pedagogic human's demonstrations. When grass is ok, the literal human is equally likely to walk on the grass or pavement. On the other hand, when grass is ok, the pedagogic human always walks on the grass in order to signal that grass is ok. (b) The beliefs of the literal robot LR and pedagogic robot PR after observing a demonstration. LR and PR assume human is literal and pedagogic, respectively. LR and PR differ in how strong their beliefs are after witnessing the human walk on pavement, leading to different problems when the human is misspecified. If the human is literal, but the robot is pedagogic, then it makes too strong of an inference. On the other hand, if the human is pedagogic, but the robot is literal, then it makes too weak of an inference.
 (a) (CIRL) the robot acts in the environment. The game's joint payoff for the robot and human is the value (expected sum of rewards) attained by the robot. (b) (CI) the robot outputs an estimate of the reward function. The game's joint payoff for the robot and human is a measure of the accuracy of the robot's inference.
 Figure 2 : 2 Figure 2: (a) The instructions given to participants in the literal and pedagogic condition, and a sample demonstration from both conditions. In the pedagogic case, participants were more likely to visit multiple safe colors, as well as to loop over safe tiles multiple times. (b) All possible reward functions. Each tile color can be either safe (0 points) or dangerous (-2 points). Figure modified from Ho et al. (2018).
 Figure 3 : 3 Figure 3: The accuracy of the robot's inferred reward for different pairs of human/robot pairs under the demonstrations provided by actual humans.
 Figure 4 : 4 Figure 4: The accuracy of the robot's inferred reward for different pairs of human/robot pairs under the demonstrations provided by humans simulated according to the human models in Section 3.2.
 Figure 5 : 5 Figure 5: The performance of the literal and pedagogic robot when demonstrations are simulated according to the demonstration mixture model (left/a) or the action mixture model (right/b).
 Figure 6 : 6 Figure6: (left/a) In blue is the mean negative log-likelihood of the pedagogic human demonstrations under the action mixture model with probability α. All individuals are assumed to have the same value of α. Note that a mixture between the literal and pedagogic model is far better than either the literal model (α = 0) or the pedagogic model (α = 1). In purple is the accuracy of the robot's inferred reward, when given demonstrations from pedagogic AH, if it assumes the human acts according to the action mixture model. (right/b) The mixture probability α that maximized log-likelihood for each individual pedagogic AH.
			 A game in which all agents have the same payoff.
			 In fact, in Demonstration-CIRL (Example 2.1), the best that the robot can do is infer a posterior distribution over rewards from the human's demonstrations, and then act optimally with respect to the posterior mean.3 Wang et al. (2019)  show that fixed-point iteration converges for all discrete distributions. For simplicity, we have written equations (1) and (2) assuming that the human and robot's prior over rewards and demonstrations is uniform, but in general, the equations can incorporate any prior (Yang et al., 2018) .
			 . The literal human H 0 is noisily-optimal with respect to the reward function r. The probability4  There is nothing that requires the literal level to be at recursion level 0 and the pedagogic level to be at recursion level 1. Our proof of Claim 2.1 holds when the literal level is any k ≥ 0 and the pedagogic level is k + 1.
			 All error bars and confidence bands in the paper depict bootstrapped 95% confidence intervals.


Abstract: In formal verification, a designer first constructs a model, with mathematically precise semantics, of the system under design, and performs extensive analysis with respect to correctness requirements. The appropriate mathematical model for embedded control systems is hybrid systems that combines the traditional state-machine based models for discrete control with classical differential-equations based models for continuously evolving physical activities. In this article, we briefly review selected existing approaches to formal verification of hybrid systems, along with directions for future research.

Formal Verification of Hybrid Systems

REVIEW OF CURRENT APPROACHES Model-based design offers a promising approach for detecting and correcting errors in early stages of system design  [33, 37, 49] . In this methodology, a designer first constructs a model, with mathematically precise semantics, of the system under design, and performs extensive analysis with respect to correctness requirements before generating the implementation from the model. Embedded systems, such as controllers in automotive, medical, and avionic systems, consist of a collection of interacting software modules reacting to a continuously evolving environment. The appropriate mathematical model for design of embedded control systems is hybrid systems that combines the traditional models for discrete behavior with classical differential-and algebraicequations based models for dynamical systems. Such models can capture both the controller -the system under design, and the plant -the environment with continuously evolving physical activities in which the system operates. Given that (1) automated verification tools have recently been successful in finding bugs in "real-world" hardware protocols and device drivers  [13, 15] , (2) tools such as Stateflow/Simulink are commonly used in automotive and avionics industry for modeling, and (3) high assurance is a necessity in safetycritical applications that deploy embedded software, formal verification of hybrid systems has been a vibrant research area for the past 20 years. This article is an overview of current research directions, aimed at providing an introductory "roadmap" rather than a comprehensive survey.
 Modeling In early 1990s, formal models for discrete reactive systems were integrated with models for dynamical systems  [2, 41] . The model of hybrid automata  [1, 2, 29]  has emerged to be a popular choice. A hybrid automaton is an extended finitestate machine whose state consists of a mode that ranges over finitely many discrete values and a finite set of realvalued variables. Each mode is annotated with constraints that specify the continuous evolution of the system, and edges between modes are annotated with guards and updates that specify discrete transitions. For example, the behavior of a self-regulating thermostat can be described by a hybrid automaton with two modes, on and off, and a single real-valued variable T modeling the temperature. To specify how the temperature changes in the mode on, we can annotate the mode with a differential equation, say, Ṫ = k(70 − T ), for a constant parameter k. Alternatively, we can use a differential inequality, say, k 1 ≤ Ṫ ≤ k2, for two constants k 1 and k2, to specify the dynamics approximately using only bounds on the derivative. Associating the mode on with an invariant constraint T ≤ 70 specifies that the system must exit this mode before the temperature exceeds 70. An edge from the mode on to the mode off with the guard T ≥ 68 specifies that, whenever the temperature exceeds 68, the system can discretely change its state by updating the mode to off. A hybrid automaton can naturally be interpreted as an infinite-state transition system, and this forms the basis for formalizing classical notions such as safety verification, property-preserving abstractions, and simulation relations, for hybrid systems. Hybrid automata are analogs of state machines, with little support for structured descriptions, and consequently, a number of formalisms have been proposed to facilitate modular descriptions of complex systems. These include modeling environments such as Shift  [17]  and Ptolemy  [19]  for hierarchical specifications of hybrid behavior; models such as hybrid I/O automata  [40] , hybrid modules  [6] , and Charon  [3] , for compositional treatment of concurrent hybrid behavior; and differential dynamic logic for logic-based specification and compositional analysis of sequential hybrid behavior  [43] . The commercial modeling tools such as Stateflow/Simulink (see www.mathworks.com) and Modelica (see www.modelica. org) are routinely used in a wide range of industries. Conceptually, it should be possible to compile models expressed in such tools into formal notations such as hybrid automata. This has turned out to be difficult in practice due to the richness of features in commercial tools and the lack of a standardized formal semantics of such features (see  [35]  for efforts aimed at semantics-preserving transformations across modeling notations).
 Symbolic reachability analysis In the safety verification problem for hybrid systems, we are given a hybrid systems model M , a set I of initial states of M , and a set S of "safe" states of M , and we want to check whether every execution of M starting in an initial state always stays within the set of safe states, and if not, report a violating execution as a counter-example. For instance, given the hybrid systems model of a collision avoidance protocol, we want to check whether the distance between two vehicles stays greater than the safety threshold for given initial conditions. Let us first note that the safety verification problem of hybrid systems cannot be solved algorithmically: with the exception of classes that severely restrict the allowed dynamics of real-valued variables such as timed automata  [5]  and initialized rectangular automata  [32] , the safety verification problem is undecidable  [1, 32] . Symbolic reachability algorithms for safety verification try to compute the set R of reachable states of a hybrid system M in an iterative manner starting from the set I of initial states. The algorithm checks, at every step of the iteration, if the current set R of reachable states is a subset of the set S of safe states, and if not, it terminates with a counter-example. In general, there is no termination guarantee, as the algorithm may keep adding more and more states to R without being able to deduce that the system is safe. The key challenge to efficient implementation is to identify a suitable representation for the set of states that supports the operations used by the iterative reachability computation. The tool HyTech was the first model checker to implement symbolic reachability analysis for hybrid systems  [7, 31] . For a hybrid automaton with n real-valued variables, each reachable set is represented by associating a finite union of ndimensional polyhedra with each mode, where a polyhedron is represented as a conjunction of linear inequalities over variables. Such a polyhedra-based representation is appealing due to the use of polyhedra in many computing applications and the availability of open-source libraries for manipulating them (c.f.  [12] ). The models are restricted to the class of linear hybrid automata (LHA): guards, updates, and invariants involve only linear expressions, and the dynamics is specified using differential inequalities that are linear constraints over first-order derivatives. For example, the LHA-admissible dynamics ẋ = ẏ ∧ 1 ≤ ẋ ≤ 2 describes two-dimensional motion along the diagonal line with bounds on speed. For LHA, the polyhedral representation is closed under both discrete transitions corresponding to mode-switches and continuous evolution according to differential constraints in a mode: given a polyhedron R describing the set of current states, the set R of states that the system can reach after a discrete mode-switch to a mode m, is a polyhedron that can be computed effectively from R, and the set R of states that the system can reach as a result letting it evolve continuously according to the dynamics associated with the mode m, is also a polyhedron that can be computed effectively from R . The most commonly used dynamics in mathematical design of control systems involves linear differential equations: if x represents the vector of state variables, u represents the vector of input variables, a linear system is described by the equation ẋ = Ax + Bu, where A and B are matrices of appropriate dimensions. Such dynamics is not allowed in LHA. Linear hybrid systems (LHS) refers to the class of hybrid automata where guards, updates, and invariants involve only linear expressions, and the dynamics is specified using linear differential equations. First note that the polyhedral representation is not closed under continuous evolution specified by linear differential equations: if x = 1 is the initial state and the dynamics is given by the linear differential equation ẋ = x, the set of reachable states is the exponential curve given by e t , for real numbers t ≥ 0. Since manipulating transcendental functions is computationally difficult, a popular strategy, first advocated by the tool Checkmate  [14] , and later refined by the tool d/dt  [11] , is to compute overapproximations of reachable sets using polyhedral representations. Given a polyhedron R representing the set of current states, to compute the set R of states that are reachable within a fixed time horizon Δ according to a given linear differential equation, the algorithm implements the following strategy. For every corner vertex v of R, it first computes the state v of the system at time Δ starting in state v. Then it computes the convex hull R 1 of all the corner vertices v of R and their respective images v . We are guaranteed that all the system trajectories start in the polyhedron R 1 and end in R 1 at time Δ, but are not necessarily inside R1 at intermediate times. The final step involves computing the polyhedron R 2 obtained by "face-lifting" R1: the number and normal vectors for facets of R 2 coincide with R1, but the facets are shifted outwards so as to include all reachable states upto time Δ. All these steps can be efficiently implemented for linear systems, and the resulting polyhedron R 2 is guaranteed to be a superset of the desired set R . The process can be repeated for successive time intervals, and the resulting approximation of the reachable set is called the flowpipe approximation. The complexity of operations on polyhedra is exponential in the number of dimensions (that is, the number of real-valued variables of the hybrid system), and since such operations are invoked repeatedly in symbolic analysis based on polyhedral representation, a significant body of work has been aimed at battling this complexity and/or replacing polyhedra with alternative representations  [11, 21, 36, 42] . Representation using zonotopes and support functions  [22, 26]  has so far resulted in the most scalable approach for the analysis of linear hybrid systems, leading to the tool SpaceEx that is able to analyze, for instance, a complex 28-dimensional helicopter controller  [20] .
 Deductive verification In deductive verification, a designer interacts with a mechanized theorem prover to generate proofs of correctness of systems. For safety verification of discrete systems, a classical proof principle relies on the notion of inductive invariants: to show that all executions of a system M starting in an initial set I stay within a safe set S, we identify a state property ϕ such that (1) all initial states satisfy ϕ; (2) ϕ is a subset of the desired safety property S; and (3) the property ϕ is preserved locally across system transitions (that is, no transition of the system changes the value of ϕ from 1 to 0). In interactive verification, the user proposes a property ϕ, and the analysis tool checks if ϕ is an inductive invariant. The concept of inductive invariants for discrete systems has been generalized and adopted to continuous-time dynamical (and hybrid) systems. We will informally explain the first such notion, called barrier certificates  [47, 48] . To show that a dynamical system M with dynamics ẋ = f (x) with initial set I satisfies a safety property S, we identify a function ψ from the states to reals such that (1) in all initial states, the value of ψ is nonnegative; (2) in all unsafe states (that is, states not in the safe set S), the value of ψ is negative; and (3) the Lie derivate of ψ with respect to the vector field f is positive on the boundary set (called the barrier) characterized by ψ(x) = 0. The first two conditions ensure that the barrier separates the initial states from the unsafe states, and the third condition ensures that system trajectories cannot escape the barrier from inside as at the barrier the flow field points inwards. Together, these conditions imply that ψ(x) ≥ 0 is an inductive invariant of the system. Typically, the desired function ψ is a polynomial function of the system variables. It is also worth noting that the barrier certificates are closely related to the notion of Lyapunov certificates for stability in classical control theory. The notion of differential invariants generalizes barrier certificates  [44] : it relaxes the third condition, and also allows more general forms of logical assertions as potential invariants. Verification of safety properties based on identifying inductive invariants avoids the iterative calculation of reachable state sets and is not limited to linear systems. The tool KeYmaera offers support to prove correctness of hybrid systems using deductive verification  [43, 44] . To check whether a given polynomial certificate satisfies all the conditions necessary for it to be a barrier certificate, the tool needs to perform symbolic differentiation, and calculations such as simplification and quantifier elimination, with formulas in the theory of reals with arithmetic operators. To fully automate deductive verification, such a tool needs to automatically generate candidates for inductive invariants, and this remains an active area of research (see  [28, 50]  for automatic generation of invariants by instantiating templates and  [45]  for generating invariants by fixpoint computation).
 Abstraction An abstraction A of a model M is a "simplified" model obtained from M such that proving safety and temporal properties of A is a sufficient condition for proving the corresponding properties of M . Abstraction is an effective strategy for scalability of verification tools, provided there is a way to compute A from M in a tool-supported manner, and a way to refine the current abstraction A if it is not adequate to prove the desired properties. In the case of hybrid systems, the simplicity of the abstract model A can be of various forms: A can be discrete while M is continuous; A can have linear dynamics while M has non-linear dynamics; and A can be a linear model of dimensionality lower than that of M . There is an extensive literature on automatic abstraction of hybrid systems. We note three representative examples. We have already seen that the dynamics admissible in the model of linear hybrid automata is simple enough to permit exact computation of reachable states using polyhedral representation. In phase portrait approximation  [30] , the dynamics ẋ = f (x) in a mode m of a hybrid system is replaced by l ≤ ẋ ≤ u, where the vectors l and u represent the lower and upper bounds on the function f over the range of values specified by the invariant constraint associated with the mode m. This clearly yields an over-approximation of the allowed system trajectories in each mode. The error introduced by the approximation can be reduced if we split the mode m into submodes, each corresponding to a different region of the state-space. Predicate abstraction is a powerful technique for extracting finite-state models from complex, potentially infinite-state, systems, and has been extended and adopted for hybrid systems  [4, 16] . In this approach, the input to the verification tool consists of a linear hybrid system, the safety property to be verified, and a finite set of Boolean predicates over system variables to be used for abstraction. An abstract state is a valid combination of truth values to the Boolean predicates, and thus, corresponds to a polyhedral set of the concrete state-space. The verifier performs an on-the-fly search of the abstract system by symbolic manipulation of polyhedra, where the computation of continuoustime successors of abstract states can be performed using flow-pipe approximations. The key computational benefit is that the continuous reachability computation is applied only to an abstract state, instead of intermediate sets of arbitrary complexity generated during iterative computation. If the initial choice of predicates is too coarse, the search finds abstract counter-examples that are infeasible in the original hybrid system, and such counter-examples can be analyzed to discover new predicates that will rule out related spurious counter-examples. A classical notion of equivalence of nondeterministic systems is simulation: a relation between states of two systems is a simulation relation, if (1) two related states have identical observations, and (2) whenever two states are related, for every transition from the first state, there exists a matching transition from the second state such that the targets of the transitions remain related. For simpler classes of hybrid sys-tems such as timed automata and O-minimal systems, one can algorithmically compute the maximal simulation relation over the states of a given system, and use the discrete quotient with respect to this relation as the abstract system which can replace the original system for verification purposes  [8] . In the context of hybrid systems, since states contain real-valued vectors, there is a natural metric over states, and this can be used to also define a coarser notion of simulation called approximating simulations: an approximating simulation relation with parameter ε requires observations of two related states to be ε-close of one another, and transitions can be matched step-by-step by staying ε-close  [23] . The resulting theory of approximating relations leads to algorithms for constructing lower-dimensional abstractions of linear systems  [52] .
 EMERGING RESEARCH DIRECTIONS Automated verification is a computationally intractable problem. Consequently, even though there are many demonstrations of interesting analyses using tools for verification of hybrid systems, scalability remains a challenge, and a significant fraction of the current research is aimed at addressing this challenge. A complementary challenge is to integrate verification tools and techniques in the design flow so as to improve the overall system reliability. We conclude this article by discussing some promising research directions towards this goal.
 Symbolic simulation In simulation, a possible execution of the model upto a finite time horizon is obtained using numerical methods, and this is a well-accepted industrial practice. A single simulation corresponds to a specific choice of inputs. A promising idea is to analyze a simulation trace corresponding to a particular choice of inputs using symbolic analysis techniques to compute the space of inputs that are close enough to the chosen one so that no inputs from this space need to be considered for subsequent simulations (see  [9, 18, 34]  for recent efforts). This integration of simulation and symbolic analysis can lead to improved coverage, and is similar to concolic testing which has proved to be effective in debugging of large-scale software systems  [24] . An added benefit is that such an approach can be implemented directly within the native simulation engine of an industrial-strength modeling environment such as Simulink/Stateflow.
 Synthesis Historically, synthesis refers to the process of computing an implementation (the "how") from a specification of the desired behavior and performance (the "what") and the assumptions on the environment (the "where"). In the more recent view, the synthesis tool facilitates the design by consistently integrating different views: a designer expresses her insights about the design using synthesis artifacts of different kinds such as models that may contain ambiguities and declarative specifications of high-level requirements, and the synthesis tool composes these different views about the structure and functionality of the system into a unified concrete implementation using a combination of algorithmic techniques. Illustrative examples of this new view of synthesis include programming by examples for spreadsheet transformations in Microsoft Excel  [27] , and sketching of bit-streaming programs using program skeletons  [51] . We believe that these ideas emerging in the programming languages community should be explored in the context of design of hybrid systems to integrate synthesis in a pragmatic way in the design cycle (see  [53]  for recent work on synthesizing switching conditions in hybrid automata).
 From models to code Generating embedded software directly from high-level models, such as hybrid systems, is appealing, but challenging due to the wide gap between the two. In current practice, this gap is bridged with significant manual effort by exploiting the run-time support offered by operating systems for managing tasks and interrupts. A key challenge to systematic software generation from hybrid models is to ensure that one can infer properties of the software from the properties of the model, and this problem is receiving increasing attention from researchers. Sample research directions include integration of control and scheduling  [10]  and static analysis of errors introduced by finite precision computations  [25] .
 Industrial applications The value of formal modeling and verification on industrially relevant problems has been demonstrated on a number of case studies. Examples of these include design and analysis of vehicle platooning protocols  [17] , identification of optimal tolerances for audio control protocol  [31] , safety verification of collision avoidance protocol for aircrafts  [46, 54] , and verification of adaptive cruise control  [39] . Yet, the level of commitment from embedded software industry remains limited to exploratory projects in collaboration with academic researchers. This is in contrast to, say, Intel's investment in formal hardware verification and Microsoft's investment in static analysis of software, which can be attributed to the identification of specific classes of errors that can be largely eliminated using verification techniques (for example, deadlocks in cache coherence protocols and misuse of API rules by third-party device driver code). Thus, a key challenge for research in formal verification of hybrid systems is to identify a compelling class of errors that designers routinely make and can be eliminated using verification techniques. An alternative path to industrial adoption is to integrate verification tools in the certification process, and this seems plausible in safety-critical domains such as software for medical devices  [38] . 			 Authorized licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 00:46:37 UTC from IEEE Xplore. Restrictions apply.
			 licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 00:46:37 UTC from IEEE Xplore. Restrictions apply.