Abstract: Since its inception, arti cial intelligence has relied upon a theoretical foundation centred around perfect rationality as the desired property o f i n telligent systems. We argue, as others have done, that this foundation is inadequate because it imposes fundamentally unsatis able requirements. As a result, there has arisen a wide gap between theory and practice in AI, hindering progress in the eld. We propose instead a property called bounded optimality. Roughly speaking, an agent is bounded-optimal if its program is a solution to the constrained optimization problem presented by its architecture and the task environment. We show h o w to construct agents with this property for a simple class of machine architectures in a broad class of real-time environments. We illustrate these results using a simple model of an automated mail sorting facility. We also de ne a weaker property, asymptotic bounded optimality ABO, that generalizes the notion of optimality in classical complexity theory. We then construct universal ABO programs, i.e., programs that are ABO no matter what real-time constraints are applied. Universal ABO programs can be used as building blocks for more complex systems. We conclude with a discussion of the prospects for bounded optimality as a theoretical basis for AI, and relate it to similar trends in philosophy, economics, and game theory.

Provably Bounded-Optimal Agents

Introduction Since before the beginning of arti cial intelligence, philosophers, control theorists and economists have l o o k ed for a satisfactory de nition of rational behaviour. This is needed to underpin theories of ethics, inductive learning, reasoning, optimal control, decision-making, and economic modelling.  Doyle 1983  has proposed that AI itself be de ned as the computational study of rational behaviour|e ectively equating rational behaviour with intelligence. The role of such de nitions in AI is to ensure that theory and practice are correctly aligned. If we de ne some property P, then we hope to be able to design a system that provably possesses property P. Theory meets practice when our systems exhibit P in reality. F urthermore, that they exhibit P in reality should be something that we actually care about. In a sense, the choice of what P to study determines the nature of the eld. There are a number of possible choices for P: Perfect rationality: the classical notion of rationality in economics and philosophy. A perfectly rational agent acts at every instant i n s u c h a w ay as to maximize its expected utility, given the information it has acquired from the environment. Since action selection requires computation, and computation takes time, perfectly rational agents do not exist for non-trivial environments. Calculative rationality: the notion of rationality studied in AI. A calculatively rational agent e v entually returns what would have been the rational choice at the beginning of its deliberation. There exist systems such as in uence diagram evaluators that exhibit this property for a decision-theoretic de nition of rational choice, and systems such as nonlinear planners that exhibit it for a logical de nition of rational choice. This is assumed to be an interesting property for a system to exhibit since it constitutes an in-principle" capacity to do the right thing. Calculative rationality is of limited value in practice, because the actual behaviour exhibited by such systems is absurdly far from being rational; for example, a calculatively rational chess program will choose the right m o ve, but may take 1 0 50 times too long to do so. As a result, AI systembuilders often ignore theoretical developments, being forced to rely on trial-and-error engineering to achieve their goals. Even in simple domains such a s c hess, there is little theory for designing and analysing high-performance programs. Metalevel rationality: a natural response to the problems of calculative rationality. A metalevel rational system optimizes over the object-level computations to be performed in the service of selecting actions. In other words, for each decision it nds the optimal combination of computation-sequence-plus-action, under the constraint that the action must be selected by the computation. Full metalevel rationality i s seldom useful because the metalevel computations themselves take time, and the metalevel decision problem is often more di cult than the object-level problem. Simple approximations to metalevel rationality h a ve proved useful in practice|for example, metalevel policies that limit lookahead in chess programs|but these engineering expedients merely serve to illustrate the lack of a theoretical basis for agent design. Bounded optimality: a bounded optimal agent behaves as well as possible given its computational resources. Bounded optimality speci es optimal programs rather than optimal actions or optimal computation sequences. Only by the former approach can we a void placing constraints on intelligent agents that cannot be met by a n y program. Actions and computations are, after all, generated by programs, and it is over programs that designers have control. We make three claims: 1. A system that exhibits bounded optimality is desirable in reality. 2. It is possible to construct provably bounded optimal programs. 3. Arti cial intelligence can be usefully characterized as the study of bounded optimality, particularly in the context of complex task environments and reasonably powerful computing devices. The rst claim is unlikely to be controversial. This paper supports the second claim in detail. The third claim may, o r m a y not, stand the test of time. We begin in section 2 with a necessarily brief discussion of the relationship between bounded optimality and earlier notions of rationality. W e note in particular that some important distinctions can be missed without precise de nitions of terms. Thus in section 3 we provide formal de nitions of agents, their programs, their behaviour and their rationality. Together with formal descriptions of task environments, these elements allow u s t o p r o ve that a given agent exhibits bounded optimality. Section 4 examines a class of agent architectures for which the problem of generating bounded optimal con gurations is e ciently soluble. The solution involves a class of interesting and practically relevant optimization problems that do not appear to have been addressed in the scheduling literature. We illustrate the results by showing how the throughput of an automated mail-sorting facility might be improved. Section 5 initiates a discussion of how bounded optimal con gurations might be learned from experience in an environment. In section 6, we de ne a weaker property, asymptotic bounded optimality ABO, that may be more robust and tractable than the strict version of bounded optimality. In particular, we can construct universal ABO programs. A program is universally ABO if it is ABO regardless of the speci c form of time dependence of the utility function.  1  Universal ABO programs can therefore be used as building blocks for more complex systems. We conclude with an assessment of the prospects for further development of this approach to arti cial intelligence.
 Historical Perspective The classical idea of perfect rationality, which developed from Aristotle's theories of ethics, work by Arnauld and others on choice under uncertainty, and Mill's utilitarianism, was put on a formal footing in decision theory by  Ramsey 1931 and vonNeumann and Morgernstern 1947.  It stipulates that a rational agent always act so as to maximize its expected utility. The expectation is taken according to the agent's own beliefs; thus, perfect rationality d o e s not require omniscience. In arti cial intelligence, the logical de nition of rationality, known in philosophy as the practical syllogism", was put forward by  McCarthy 1958 , and reiterated strongly by  Newell 1981 . Under this de nition, an agent should take a n y action that it believes is guaranteed to achieve a n y of its goals. If AI can be said to have had a theoretical foundation, then this de nition of rationality has provided it. McCarthy believed, probably correctly, that in the early stages of the eld it was important to concentrate on epistemological adequacy" before heuristic adequacy" | that is, capability in principle rather than in practice. The methodology that has resulted involves designing programs that exhibit calculative rationality, and then using various speedup techniques and approximations in the hope of getting as close as possible to perfect rationality. Our belief, albeit unproven, is that the simple agent designs that ful ll the speci cation of calculative rationality m a y not provide good starting points from which to approach bounded optimality. Moreover, a theoretical foundation based on calculative rationality cannot provide the necessary guidance in the search. It is not clear that AI would have e m barked on the quest for calculative rationality had it not been operating in the halcyon days before formal intractability results were discovered. One response to the spectre of complexity has been to rule it out of bounds.  Levesque and Brachman 1987  suggest limiting the complexity of the environment so that calculative and perfect rationality coincide. Doyle and Patil 1991 argue strongly against this position. Economists have used perfect rationality as an abstract model of economic entities, for the purposes of economic forecasting and designing market mechanisms. This makes it possible to prove theorems about the properties of markets in equilibrium. Unfortunately, as Simon 1982 pointed out, real economic entities have limited time and limited powers of deliberation. He proposed the study of bounded r ationality, i n vestigating : : : the shape of a system in which e ectiveness in computation is one of the most important w eapons of survival." Simon's work focussed mainly on satis cing designs, which deliberate until reaching some solution satisfying a preset aspiration level." The results have descriptive v alue for modelling various actual entities and policies, but no general prescriptive framework for bounded rationality w as developed. Although it proved possible to calculate optimal aspiration levels for certain problems, no structural variation was allowed in the agent design. In the theory of games, bounds on the complexity of players have become a topic of intense interest. For example, it is a troubling fact that defection is the only equilibrium strategy for unbounded agents playing a xed number of rounds of the Prisoners' Dilemma game. Neyman's theorem  Neyman, 1985 , recently proved by P apadimitriou and Yannakakis 1994, shows that an essentially cooperative equilibrium exists if each agent i s a nite automaton with a number of states that is less than exponential in the number of rounds. This is essentially a bounded optimality result, where the bound is on space rather than on speed of computation. This type of result is made possible by a shift from the problem of selecting actions to the problem of selecting programs. I. J. Good 1971 distinguished between perfect or type I" rationality, and metalevel or type II" rationality. He de nes this as the maximization of expected utility taking into account deliberation costs." Simon 1976 also says: The global optimization problem is to nd the least-cost or best-return decision, net of computational costs." Although type II rationality seems to be a step in the right direction, it is not entirely clear whether it can be made precise in a way that respects the desirable intuition that computation is important. We will try one interpretation, although there may be others.  2  The key issue is the space over which the maximization" or optimization" occurs. Both Good and Simon seem to be referring to the space of possible deliberations associated with a particular decision. Conceptually, there is an object-level machine" that executes a sequence of computations under the control of a meta-level machine." The outcome of the sequence is the selection of an external action. An agent exhibits type II rationality if at the end of its deliberation and subsequent action, its utility is maximized compared to all possible deliberate act pairs in which it could have engaged. For example, Good discusses one possible application of type II rationality i n c hess programs. In this case, the object-level steps are node expansions in the game tree, followed by backing up of leaf node evaluations to show the best move. For simplicity w e will assume a per-move time limit. Then a type II rational agent will execute whichever sequence of node expansions chooses the best move, of all those that nish before the time limit.  3  Unfortunately, the computations required in the metalevel machine" to select the object-level deliberation may be extremely expensive. Good actually proposes a fairly simple and nearly practical metalevel decision procedure for chess, but it is far from optimal. It is hard to see how a t ype II rational agent could justify executing a suboptimal object-level computation sequence if we limit the scope of the optimization problem to a single decision. The di culty can only be resolved by thinking about the design of the agent program, which generates an unbounded set of possible deliberations in response to an unbounded set of circumstances that may arise during the life of the agent. Philosophy has also seen a gradual evolution in the de nition of rationality. There has been a shift from consideration of act utilitarianism | the rationality of individual acts | to rule utilitarianism, or the rationality of general policies for acting. This shift has been caused by di culties with individual versus societal rationality, rather than any consideration of the di culty of computing rational acts. Some consideration has been given more recently to the tractability of general moral policies, with a view to making them understandable and usable by persons of average intelligence  Brandt, 1953 . Cherniak 1986  has suggested a de nition of minimal rationality", specifying lower bounds on the reasoning powers of any rational agent, instead of upper bounds. A philosophical proposal generally consistent with the notion of bounded optimality can be found in Dennett's Moral First Aid Manual" 1986. Dennett explicitly discusses the idea of reaching equilibrium within the space of decision procedures. He uses as an example the PhD admissions procedure of a philosophy department. He concludes, as do we, that the best procedure may be neither elegant nor illuminating. The existence of such a procedure, and the process of reaching it, are the main points of interest. Many researchers in AI, some of whose work is discussed below, have w orked on the problem of designing agents with limited computational resources. The 1989 AAAI Symposium on AI and Limited Rationality F ehling & Russell, 1989 contains an interesting variety o f w ork on the topic. Much of this work is concerned with metalevel rationality. Metareasoning | reasoning about reasoning | is an important technique in this area, since it enables an agent to control its deliberations according to their costs and bene ts. Combined with the idea of anytime  Dean & Boddy, 1988  or exible algorithms  Horvitz, 1987 , that return better results as time goes by, a simple form of metareasoning allows an agent to behave w ell in a real-time environment. A simple example is provided by iterative-deepening algorithms used in game-playing.  Breese and Fehling 1990  apply similar ideas to controlling multiple decision procedures. Russell and Wefald 1989 give a general method for precompiling certain aspects of metareasoning so that a system can eciently estimate the e ects of individual computations on its intentions, giving ne-grained control of reasoning. These techniques can all be seen as approximating metalevel rationality; they provide useful insights into the general problem of control of reasoning, but there is no reason to suppose that the approximations used are optimal in any sense. The intuitive notion of bounded optimality seems to have become current in the AI community in the mid-1980's. Horvitz 1987 uses the term bounded optimality to refer to the optimization of computational utility given a set of assumptions about expected problems and constraints in reasoning resources.  " Russell and Wefald 1991  say that an agent exhibits bounded optimality for a given task environment if its program is a solution to the constrained optimization problem presented by its architecture." Recent w ork by  Etzioni 1989 and Zilberstein 1991  can be seen as optimizing over a wellde ned set of agent designs, thereby making the notion of bounded optimality more precise. In the next section, we build a suitable set of general de nitions from the ground up, so that we can begin to demonstrate examples of provably bounded optimal agents.
 Agents, Architectures and Programs Intuitively, a n agent is just a physical entity that we wish to view in terms of its perceptions and actions. What counts in the rst instance is what it does, not necessarily what it thinks, or even whether it thinks at all. This initial refusal to consider further constraints on the internal workings of the agent such as that it should reason logically, for example helps in three ways: rst, it allows us to view such cognitive faculties" as planning and reasoning as occurring in the service o f nding the right thing to do; second, it makes room for those among us  Agre & Chapman, 1987; Brooks, 1986  who take the position that systems can do the right thing without such cognitive faculties; third, it allows more freedom to consider various speci cations, boundaries and interconnections of subsystems. We begin by de ning agents and environments in terms of the actions and percepts that they exchange, and the sequence of states they go through. The agent is described by a n agent function from percept sequences to actions. This treatment is fairly standard see, e.g.,  Genesereth & Nilsson, 1987.  We then go inside" the agent t o l o o k a t t h e agent program that generates its actions, and de ne the implementation" relationship between a program and the corresponding agent function. We consider performance measures on agents, and the problem of designing agents to optimize the performance measure.
 Specifying agents and environments An agent can be described abstractly as a mapping the agent function from percept sequences to actions. Let O be the set of percepts that the agent can receive a t a n y instant, and A be the set of possible actions the agent can carry out in the external world. Since we are interested in the behaviour of the agent o ver time, we i n troduce a set of time points or instants, T. The set T is totally ordered by the relation with a unique least element. Without loss of generality, w e let T be the set of non-negative i n tegers. The percept history of an agent is a sequence of percepts indexed by time. We de ne the set of percept histories to be O T = fO T : T ! Og. The pre x of a history O T 2 O T till time t is denoted O t and is the projection of O T on 0::t . We can de ne the set of percept history pre xes as O t = fO t j t 2 T; O T 2 O T g. Similarly, w e de ne the set of action histories A T = fA T : T ! Ag. The set of action history pre xes is A t , de ned as the set of projections A t of histories A T 2 A T . De nition 1 Agent function: a mapping f : O t ! A where A T t = fO t Note that the agent function is an entirely abstract entity, unlike the agent program that implements it. Note also that the output" of the agent function for a given percept sequence may b e a n ull action, for example if the agent is still thinking about what to do. The agent function speci es what the agent d o e s a t e a c h time step. This is crucial to the distinction between perfect rationality and calculative rationality. Agents live i n e n vironments. The states of an environment E are drawn from a set X. The set of possible state trajectories is de ned as X T = fX T : T ! Xg. The agent does not necessarily have full access to the current state X T t, but the percept received by the agent does depend on the current state through the perceptual ltering function f p . The e ects of the agent's actions are represented by the environment's transition function f e , which speci es the next state given the current state and the agent's action. An environment i s therefore de ned as follows: De nition 2 Environment E: a set of states X with a distinguished initial state X 0 , a transition function f e and a perceptual lter function f p such that X T 0 = X 0 X T t + 1 = f e A T t; X T t O T t = f p X T t The state history X T is thus determined by the environment and the agent function. We use the notation e ectsf;E to denote the state history generated by an agent function f operating in an environment E. We will also use the notation E;A t to denote the state history generated by applying the action sequence A t starting in the initial state of environment E. Notice that the environment is discrete and deterministic in this formulation. We can extend the de nitions to cover non-deterministic and continuous environments, but at the cost of additional complexity in the exposition. None of our results depend in a signi cant way on discreteness or determinism.
 Specifying agent implementations We will consider a physical agent as consisting of an architecture and a program. The architecture is responsible for interfacing between the program and the environment, and for running the program itself. With each architecture M, w e associate a nite programming language L M , which is just the set of all programs runnable by the architecture. An agent program is a program l 2 L M that takes a percept as input and has an internal state drawn from a set I with initial state i 0 . The initial internal state depends on the program l, but we will usually suppress this argument. The set of possible internal state histories is I T = fI T : T ! Ig. The pre x of an internal state history I T 2 I T till time t is denoted I t and is the projection of I T on 0::t . De nition 3 An a r chitecture M is a xed interpreter for an agent program that runs the program for a single time step, updating its internal state and generating an action: M : L M I O ! I A where hI T t + 1 ; A T ti = Ml;I T t; O T t Thus, the architecture generates a stream of actions according to the dictates of the program. Because of the physical properties of the architecture, running the program for a single time step results in the execution of only a nite number of instructions. The program may often fail to reach a decision" in that time step, and as a result the action produced by the architecture may b e n ull or the same as the previous action, depending on the program design.
 Relating agent speci cations and implementations We can now relate agent programs to the corresponding agent functions. We will say that an agent program l running on a machine M implements the agent function Agentl;M. The agent function is constructed in the following de nition by specifying the action sequences produced by l running on M for all possible percept sequences. Note the importance of the Markovian" construction using the internal state of the agent to ensure that actions can only be based on the past, not the future. De nition 4 A p r ogram l running on M implements the agent function f = Agentl;M, de ned as follows. For any environment E = X; f e ; f p , fO t = A T t where hI T t + 1 ; A T ti = Ml; I T t; O T t O T t = f p X T t X T t + 1 = f e A T t; X T t X T 0 = X 0 I T 0 = i 0 Although every program l induces a corresponding agent function Agentl;M, the action that follows a given percept is not necessarily the agent's response" to that percept; because of the delay incurred by deliberation, it may only re ect percepts occurring much earlier in the sequence. Furthermore, it is not possible to map every agent function to an implementation l 2 L M . We can de ne a subset of the set of agent functions f that are implementable on a given architecture M and language L M : FeasibleM = ff j 9 l 2 L M ; f = Agentl;Mg Feasibility is related to, but clearly distinct from, the notion of computability. Computability refers to the existence of a program that eventually returns the output speci ed by a function, whereas feasibility refers to the production of the output at the appropriate point in time. The set of feasible agent functions is therefore much smaller than the set of computable agent functions.
 Performance measures for agents To e v aluate an agent's performance in the world, we de ne a real-valued utility function U on state histories: U : X T ! The utility function should be seen as external to the agent and its environment. It de nes the problem to be solved by the designer of the agent. Some agent designs may incorporate an explicit representation of the utility function, but this is by no means required. We will use the term task environment to denote the combination of an environment and a utility function. Recall that the agent's actions drive the environment E through a particular sequence of states in accordance with the function e ectsf;E. We can de ne the value of an agent function f in the environment E as the utility of the state history it generates: V f;E = Ue ectsf;E If the designer has a set E of environments with a probability distribution p over them, instead of a single environment E, then the value of the agent i n E is de ned as the expected value over the elements of E. By a slight abuse of notation, V f;E = X E2E pEV f;E We can assign a value V l;M;E to a program l executed by the architecture M in the environment E simply by looking at the e ect of the agent function implemented by the program: V l;M;E = V Agentl;M; E = Ue ectsAgentl;M; E As above, we can extend this to a set of possible environments as follows: V l;M;E = X E2E pEV l;M;E
 Perfect rationality and bounded optimality As discussed in Section 2, a perfectly rational agent selects the action that maximizes its expected utility, given the percepts so far. In our framework, this amounts to an agent function that maximizes V f;E o ver all possible agent functions. De nition 5 A p erfectly rational agent for a set E of environments has an agent function f opt such that f opt = argmax f V f;E This de nition is a persuasive speci cation of an optimal agent function for a given set of environments, and underlies several recent projects in intelligent agent design  Dean & W ellman,1991; Doyle, 1988; Hansson & Mayer, 1989 . A direct implementation of this speci cation, which ignores the delay incurred by deliberation, does not yield a reasonable solution to our problem the calculation of expected utilities takes time for any real agent. In terms of our simple formal description of agents introduced above, it is easy to see where the di culty has arisen. In designing the agent program, logicists and decision theorists have concentrated on specifying an optimal agent function f opt in order to guarantee the selection of the best action history. The function f opt is independent of the architecture M. Unfortunately, no real program in L M implements this function in a non-trivial environment, because optimal actions cannot usually be computed before the next percept arrives. That is, quite frequently, f opt 6 2 FeasibleM. Suppose the environment consists of games of chess under tournament rules against some population of human grandmasters, and suppose M is some standard personal computer. Then f opt describes an agent that always plays in such a w ay as to maximize its total expected points against the opposition, where the maximization is over the moves it makes. We claim that no possible program can play this way. It is quite possible, using depth-rst alpha-beta search to termination, to execute the program that chooses say the optimal minimax move in each situation, but the agent function induced by this program is not the same as f opt . In particular, it ignores such percepts as the dropping of its ag indicating a loss on time. The trouble with the perfect rationality de nition arose because of unconstrained optimization over the space of f's in the determination of f opt , without regard to feasibility. Similarly, metalevel rationality assumes unconstrained optimization over the space of deliberations. To escape this quandary, w e propose a machine-dependent standard of rationality, in which w e maximize V over the implementable set of agent functions FeasibleM. That is, we impose optimality constraints on programs rather than on agent functions or deliberations. De nition 6 A b ounded-optimal agent with architecture M for a set E of environments has an agent program l opt such that l opt = argmax l2L M V l;M;E We can see immediately that this speci cation avoids the most obvious problems with Type I and Type II rationality. Consider our chess example, and suppose the computer has a total program memory of 8 megabytes. Then there are 2 2 26 possible programs that can be represented in the machine, of which a m uch smaller number play legal chess. Under tournament conditions, one or more of these programs will have the best expected performance. Each is a suitable candidate for l opt . T h us bounded optimality is, by de nition, a feasible speci cation; moreover, a program that achieves it is highly desirable. We are not yet ready to announce the identity o f l opt for chess on an eight-megabyte PC, so we will begin with a more restricted problem.
 Provably Bounded-Optimal Agents In order to construct a provably bounded optimal agent, we m ust carry out the following steps: Specify the properties of the environment in which actions will be taken, and the utility function on the behaviours. Specify a class of machines on which programs are to be run. Propose a construction method. Prove that the construction method succeeds in building bounded optimal agents. The methodology is similar to the formal analysis used in the eld of optimal control, which studies the design of controllers agents for plants environments. In optimal control theory, a controller is viewed as an essentially instantaneous implementation of an optimal agent function. In contrast, we focus on the computation time required by the agent, and the relation between computation time and the dynamics of the environment.
 Episodic, real-time task environments In this section, we will consider a restricted class of task environments which w e call episodic environments. In an episodic task environment, the state history generated by the actions of the agent can be considered as divided into a series of episodes, each of which is terminated by an action. Let A ? A be a distinguished set of actions that terminate an episode. The utility of the complete history is given by the sum of the utilities of each episode, which is determined in turn by the state sequence. After each A 2 A ? , the environment resets" to a state chosen at random from a stationary probability distribution P init . In order to include the e ects of the choice of A in the utility of the episode, we notionally divide the environment state into a con guration" part and a value" part, such that the con guration part determines the state transitions while the value part determines the utility of a state sequence. Actions in A ? reset the con guration part, while their value" is recorded in the value part. These restrictions mean that each episode can be treated as a separate decision problem, and translate into the following property: if agent program l 1 has higher expected utility on individual episodes than agent l 2 , it will have higher expected utility in the corresponding episodic task environment. A real-time task environment is one in which the utility of an action depends on the time at which it is executed. Usually, this dependence will be su ciently strong to make calculative rationality an unacceptably bad approximation to perfect rationality. An automated mail sorter 4 provides an illustrative example of an episodic task environment see Figure  1 . Such a machine scans handwritten or printed addresses zipcodes on mail pieces and dispatches them to appropriate bins. Each episode starts with the arrival of a new mail piece and terminates with the execution of the physical action recommended by the sorter: routing of the piece to a speci c bin. The con guration part" of the environment corresponds to the letter feeder side, which provides a new, randomly selected letter after the previous letter is sorted. The value part" of the state corresponds to the state of the receiving bins, which determines the utility of the process. The aim is to maximize the accuracy of sorting while minimizing the reject percentage and avoiding jams. A jam occurs if the current piece is not routed to the appropriate bin, or rejected, before the arrival of the next piece. We n o w provide formal de nitions for three varieties of real-time task environments: xed deadlines, xed time cost and stochastic deadlines. Taking an action in A ? at any time before the deadline results in the same utility: U E;A t 1 = U E;A t d ,1 2 A T 1 t where " denotes sequence c oncatenation, t t d , A T 1 t 2 A ? , and A t,1 1 and A the utilities di er by the di erence in time cost: U E;A t 2 2 = U E;A t 1 1 , ct 2 , t 1 Strictly speaking, there are no task environments with xed time cost. Utility v alues have a nite range, so one cannot continue incurring time costs inde nitely. F or reasonably short times and reasonably small costs, a linear utility penalty is a useful approximation.
 Stochastic deadlines While xed-deadline and xed-cost task environments occur frequently in the design of real-time systems, uncertainty about the time-dependence of the utility function is more common. It also turns out to be more interesting, as we see below. A stochastic deadline is represented by uncertainty concerning the time of occurrence of a xed deadline. In other words, the agent has a probability distribution p d for the deadline time t d . We assume that the deadline must come eventually, so that P t2T p d t = 1 . We also de ne the cumulative deadline distribution P d . If the deadline does not occur at a known time, then we need to distinguish between two cases: The agent receives a percept, called a herald  Dean & Boddy, 1988 , which announces an impending deadline. We model this using a distinguished percept O d : O T t d = O d If the agent responds immediately, then it meets the deadline." No such percept is available, in which case the agent i s w alking blindfolded towards the utility cli . By deliberating further, the agent risks missing the deadline but may improve its decision quality. An example familiar to most readers is that of deciding whether to publish a paper in its current form, or to embellish it further and risk being scooped." We do not treat this case in the current paper. Formally, the stochastic deadline case is similar to the xed deadline case, except that t d is drawn from the distribution p d . The utility of executing an action history pre x A t in E is the expectation of the utilities of that state history pre x over the possible deadline times. De nition 9 Stochastic deadline: A task environment class hE; U i of xed-deadline task environments has a stochastic deadline distributed a c cording to p d if, for any action history pre x A t , U E; A t = X t 0 2T p d t 0 U E t 0 ; A t where hE t 0 ; U i is a task environment in hE; U i with a xed d e adline at t 0 . The mail sorter example is well described by a stochastic deadline. The time between the arrival of mail pieces at the image processing station is distributed according to a density function p d , which will usually be Poisson.
 Agent programs and agent architecture We consider simple agent programs for episodic task environments, constructed from elements of a set R = fr 1 ; : : : ; r n g of decision procedures or rules. Each decision procedure recommends but does not execute an action A i 2 A ? , and an agent program is a xed sequence of decision procedures. For our purposes, a decision procedure is a black b o x with two parameters: a run time t i 0, which i s a n i n teger that represents the time taken by the procedure to compute an action. a quality q i 0, which is a real number. This gives the expected reward resulting from executing its action A i at the start of an episode: q i = U E;A i 1 Let M J denote an agent architecture that executes decision procedures in the language J . Let t M denote the maximum runtime of the decision procedures that can be accommodated in M. For example, if the runtime of a feedforward neural network is proportional to its size, then t M will be the runtime of the largest neural network that ts in M. The architecture M executes an agent program s = s 1 : : : s m by running each decision procedure in turn, providing the same input to each as obtained from the initial percept. When a deadline arrives at a xed time t d , or heralded by the percept O d , or when the entire sequence has been completed, the agent selects the action recommended by the highest-quality procedure it has executed: Ms; I T t d ; O T t d = hi 0 ; actionI T t d i Ms; I T t s ; O T t s = hi 0 ; actionI T t s i where t s = P s i 2s t i 2 Ms; I T t; O d = hi 0 ; actionI T ti where M updates the agent's internal state history I T t such that actionI T t is the action recommended by a completed decision procedure with the highest quality. When this action is executed, the internal state of the agent is re-initialized to i 0 . This agent design works in all three of the task environment categories described above. Next we derive the value V s; M; E of an agent program s in environment E running on M for the three real-time regimes and show h o w to construct bounded optimal agents for these task environments.
 Bounded optimality with xed deadlines From Equation  2 , we know that the agent picks the action in A ? recommended by the decision procedure r with the highest quality that is executed before the deadline t d arrives. Let s 1 : : : s j be the longest pre x of the program s such that P j i=1 t i t d . F rom De nition 7 and Equation  1 , it follows that V s; M; E = Q j 3 where Q i = maxfq 1 ; : : : ; q i g. Given this expression for the value of the agent program, we can easily show the following: Theorem 1 Let r = arg max r i 2 R;t i t d q i . The singleton sequence r is a bounded optimal program for M in an episodic task environment with a known deadline t d . That is, the best program is the single decision procedure of maximum quality whose runtime is less than the deadline.
 Bounded optimality with xed time cost From Equation  2 , we know that the agent picks the action in A ? recommended by the best decision procedure in the sequence, since M runs the entire sequence s = s 1 : : : s m when there is no deadline. From De nition 8 and Equation  1 , we h a ve V s; M; E = Q m , c m X i=1 t i 4 Given this expression for the value of the agent program, we can easily show the following: Theorem 2 Let r = arg max r i 2 R q i ,ct i . The singleton sequence r is a bounded optimal program for M in an episodic task environment with a xed time cost c. That is, the optimal program is the single decision procedure whose quality, net of time cost, is highest.
 Bounded optimality with stochastic deadlines With a stochastic deadline distributed according to p d , the value of an agent program s = s 1 : : : s m is an expectation. From De nition 9, we can calculate this as P t2T p d tV s; M; E t , where hE t ; U i is a task environment with a xed deadline at t. After substituting for V s; M; E t from Equation 3, this expression simpli es to a summation, over the procedures in the sequence, of the probability o f i n terruption after the i th procedure in the sequence multiplied by the quality of the best completed decision procedure: V s V s; M; E = m X i=1 P d P i+1 j=1 t j , P d P i j=1 t j Q i 5 where P d t = R t ,1 p d t 0 dt 0 and P d t = 1 for t P m i=1 t i . A simple example serves to illustrate the value function. Consider R = fr 1 ; r 2 ; r 3 g. The rule r 1 has a quality of 0.2 and needs 2 seconds to run: we will represent this by r 1 = 0 :2; 2. The other rules are r 2 = 0 :5; 5; r 3 = 0 :7; 7. The deadline distribution function p d is a uniform distribution over 0 to 10 seconds. The value of the sequence r 1 r 2 r 3 is V r 1 r 2 r 3 = :7 , :2 :2 + 1 , :7 :5 + 1 , 1 :  De nition 10 Performance p r o le: Fo r a s e quence s, the performance p r o le Q s t gives the quality of the action returned if the agent is interrupted a t t: Q s t = maxfq i : i X j=1 t j tg For a uniform deadline density function, the value of a sequence is proportional to the area under the performance pro le up to the last possible interrupt time. Note that the height of the pro le during the interval of length t i while rule i is running is the quality o f the best of the previous rules. From De nition 10, we h a ve the following obvious property: Lemma 1 The performance p r o le of any sequence is monotonically nondecreasing. It is also the case that a sequence with higher quality decisions at all times is a better sequence: Lemma 2 If 8t Q s 1 t Q s 2 t, then V s 1 V s 2 . In this case we s a y that Q s 1 dominates Q s 2 . We can use the idea of performance pro les to establish some useful properties of optimal sequences. Lemma 3 There exists an optimal sequence that is sorted in increasing order of q's. Without Lemma 3, there are P n i=1 i! possible sequences to consider. The ordering constraint eliminates all but 2 n sequences. It also means that in proofs of properties of sequences, we n o w need consider only ordered sequences. In addition, we can replace Q i in Equation 5 by q i . The following lemma establishes that a sequence can always be improved by the addition of a better rule at the end: Lemma 4 For every sequence s = s 1 : : : s m sorted in increasing order of quality, and single step z with q z q s m , V sz V s. Corollary 1 There exists an optimal sequence ending with the highest-quality rule in R. The following lemma re ects the obvious intuition that if one can get a better result in less time, there's no point spending more time to get a worse result: Lemma 5 There exists an optimal sequence whose rules are in nondecreasing order of t i . We n o w apply these preparatory results to derive algorithms that construct bounded optimal programs for various deadline distributions.
 General distributions For a general deadline distribution, the dynamic programming method can be used to obtain an optimal sequence of decision rules in pseudo-polynomial time. We construct an optimal sequence by using the de nition of V s; M; E in Equation  5 . Optimal sequences generated by the methods are ordered by q i , in accordance with Lemma 3. We construct the table Si; t, where each e n try in the table is the highest value of any sequence that ends with rule r i at time t. W e assume the rule indices are arranged in increasing order of quality, and t ranges from the start time 0 to the end time L = P r i 2R t i . The update rule is: Si; t = max k2 0:::i,1 Sk;t, t i + q i , q k 1 , P d t with boundary condition Si; 0 = 0 for each rule i and S0; t = 0 for each time t From Corollary 1, we can read o the best sequence from the highest value in row n of the matrix S. Theorem 3 The DP algorithm computes an optimal sequence in time On 2 L where n is the number of decision procedures in R. The dependence on L in the time complexity of the DP algorithm means that the algorithm is not polynomial in the input size. Using standard rounding and scaling methods, however, a fully polynomial approximation scheme can be constructed. Although we d o n o t have a hardness proof for the problem, John Binder 1994 has shown that if the deadline distribution is used as a constant-time oracle for nding values of Pt, any algorithm will require an exponential number of calls to the oracle in the worst case.
 Long uniform distributions If the deadline is uniformly distributed over a time interval greater than the sum of the running times of the rules, we will call the distribution a long uniform distribution. Consider the rule sequence s = s 1 : : : s m drawn from the rule set R. With a long uniform distribution, the probability that the deadline arrives during rule s i of the sequence s is independent o f the time at which s i starts. This permits a simpler form of Equation 5: V s; M; E = P m,1 i=1 P d t i+1 q i + q m 1 , P m i=1 P d t i 6 To derive an optimal sequence under a long uniform distribution, we obtain a recursive speci cation of the value of a sequence as with a 2 R and s = s 1 : : : s m being some sequence in R. V as; M; E = V s; M; E + q a P d t 1 , q m P d t a 7 This allows us to de ne a dynamic programming scheme for calculating an optimal sequence using a state function Si; j denoting the highest value of a rule sequence that starts with rule i and ends in rule j. F rom Lemma 3 and Equation  7 , the update rule is: Si; j = max i kj Sk;j + P d t k q i , P d t i q j 8 with boundary condition Si; i = 1 , P d t i q i 9 From Corollary 1, we know that an optimal sequence for the long uniform distribution ends in r n , the rule with the highest quality i n R. Thus, we only need to examine Si; n; 1 i n. Each e n try requires On computation, and there are n entries to compute. Thus, the optimal sequence for the long uniform case can be calculated in On 2 . Theorem 4 An optimal sequence o f d e cision procedures for a long uniform deadline distribution can be determined i n On 2 time where n is the number of decision procedures in R.
 Short uniform distributions When P n i=1 P d t i 1, for a uniform deadline distribution P d , w e call it short. This means that some sequences are longer than the last possible deadline time, and therefore some rules in those sequences have no possibility of executing before the deadline. For such sequences, we cannot use Equation 7 to calculate V s. However, any such sequence can be truncated by removing all rules that would complete execution after the last possible deadline. The value of the sequence is una ected by truncation, and for truncated sequences the use of Equation 7 is justi ed. Furthermore, there is an optimal sequence that is a truncated sequence. Since the update rule 8 correctly computes Si; j for truncated sequences, we can use it with short uniform distributions provided we add a check to ensure that the sequences considered are truncated. Unlike the long uniform case, however, the identity of the last rule in an optimal sequence is unknown, so we need to compute all n 2 entries in the Si; j table. Each e n try computation takes On time, thus the time to compute an optimal sequence is On 3 . Theorem 5 An optimal sequence o f d e cision procedures for a short uniform deadline distribution can be determined i n On 3 time where n is the number of decision procedures in R.
 Exponential distributions For an exponential distribution, P d t = 1 ,e , t . Exponential distributions allow an optimal sequence to be computed in polynomial time. Let p i stand for the probability that rule i is interrupted, assuming it starts at 0. Then p i = P d t i = 1 , e , t i : For the exponential distribution, V s; M; E simpli es out as: V s; M; E = m,1 X i=1 h i j=1 1 , p j i p i+1 q i + h m j=1 1 , p j i q m This yields a simple recursive speci cation of the value V as; M; E of a sequence that begins with the rule a: V as; M; E = 1 , p a p 1 q a + 1 , p a V s; M; E We will use the state function Si; j which represents the highest value of any rule sequence starting with i and ending in j. Si; j = max i kj 1 , p i p k q i + 1 , p i Sk;j with boundary condition Si; i = q i 1 , p i . For any given j, Si; j can be calculated in On 2 . From Corollary 1, we know that there is an optimal sequence whose last element i s the highest-valued rule in R. Theorem 6 An optimal sequence o f d e cision procedures for an exponentially distributed stochastic deadline can be determined i n On 2 time where n is the number of decision procedures in R. The proof is similar to the long uniform distribution case.
 Simulation results for a mail-sorter The preceding results provide a set of algorithms for optimizing the construction of an agent program for a variety of general task environment classes. In this section, we illustrate these results and the possible gains that can be realized in a speci c task environment, namely, a simulated mail-sorter. First, let us be more precise about the utility function U on episodes. There are four possible outcomes; the utility of outcome i is u i . 1. The zipcode is successfully read and the letter is sent to the correct bin for delivery. 2. The zipcode is misread and the letter goes to the wrong bin. 3. The letter is sent to the reject bin. 4. The next letter arrives before the recognizer has nished, and there is a jam. Since letter arrival is heralded, jams cannot occur with the machine architecture given in Equation  2 . Without loss of generality, w e set u 1 = 1 :0 and u 2 = 0 :0. If the probability of a rule recommending a correct destination bin is p i , then q i = p i u 1 + 1 , p i u 2 = p i . W e assume that u 2 u 3 , hence there is a threshold probability below which the letter should be sent to the reject bin instead. We will therefore include in the rule set R a rule r reject that has zero runtime and recommends rejection. The sequence construction algorithm will then automatically exclude rules with quality l o wer than q reject = u 3 . The overall utility for an episode is chosen to be a linear combination of the quality of sorting q i , the probability o f rejection or the rejection rate given by Pt 1 , where t 1 is the runtime of the rst non-reject rule executed, and the speed of sorting measured by the arrival time mean. The agent program in  Boser et al. 1992  uses a single neural network on a chip. We show that under a variety of conditions an optimized sequence of networks can do signi cantly better than any single network in terms of throughput or accuracy. W e examine the following experimental conditions: We assume that a network that executes in time t has a recognition accuracy p that depends on t. W e consider p = 1 ,e ,t . The particular choice of is irrelevant because the scale chosen for t is arbitrary. W e c hoose = 0 :9, for convenience Figure  3a . We include r reject with q reject = u 3 and t reject = 0 . We consider arrival time distributions that are Poisson with varying means. Figure  3b  shows three example distributions, for means 1, 5, and 9 seconds. We create optimized sequences from sets of 40 networks with execution times taken at equal intervals from t = 1 to 40. We compare a BO sequence: a bounded optimal sequence; b Best singleton: the best single rule; c 50 rule: the rule whose execution time is the mean of the distribution i.e., it will complete in 50 of cases; d 90 rule: the rule whose execution time guarantees that it will complete in 90 of cases. In the last three cases, we add r reject as an initial step; the BO sequence will include it automatically. We measure the utility per second as a function of the mean arrival rate Figure  4 . This shows that there is an optimal setting of the sorting machinery at 6 letters per minute inter-arrival time = 10 seconds for the bounded optimal program, given that we h a ve xed at 0.9. Finally, w e i n vestigate the e ect of the variance of the arrival time on the relative performance of the four program types. For this purpose, we use a uniform distribution centered around 20 seconds but with di erent widths to vary the variance without a ecting the mean Figure  5 . We notice several interesting things about these results: The policy of choosing a rule with a 90 probability of completion performs poorly for rapid arrival rates 3, but catches up with the performance of the best single rule for slower arrival rates 4. This is an artifact of the exponential accuracy pro le for any 0:5, where the di erence in quality of the rules with run times greater than 6 seconds is quite small. The policy of choosing a rule with a 50 probability of completion fares as well as the best single rule for very high arrival rates 2, but rapidly diverges from it thereafter, performing far worse for arrival time means greater than 5 seconds. Both the best sequence and the best single rule give their best overall performance at an arrival rate of around 6 letters per minute. The performance advantage of the optimal sequence over the best single rule is about 7 at this arrival rate. It should be noted that this is a signi cant performance advantage that is obtainable with no extra computational resources. For slower arrival rates 7, the di erence between the performance of the best rule and the best sequence arises from the decreased rejection rate of the best sequence. With the exponential accuracy pro le 0:5 the advantage of running a rule with a shorter completion time ahead of a longer rule is the ability to reduce the probability of rejecting a letter. For high arrival rates inter-arrival times of 1 to 4 seconds, it is useful to have a few short rules instead of a longer single rule. Figure  5  shows that the best sequence performs better than the best single rule as the variance of the arrival time increases.  5  The performance of the optimal sequence also appears to be largely una ected by v ariance. This is exactly the behaviour we expect to observe | the ability to run a sequence of rules instead of committing to a single one gives it robustness in the face of increasing variance. Since realistic environments can involve unexpected demands of many kinds, the possession of a variety of default behaviours of graded sophistication would seem to be an optimal design choice for a bounded agent. 5. The performance of the 50 rule is at because the uniform distributions used in this experiment h a ve xed mean and are symmetric, so that the 50 rule is always the rule that runs for 20 seconds. The 90 rule changes with the variance, and the curve exhibits some discretization e ects. These could be eliminated using a ner-grained set of rules.
 Learning Approximately Bounded-Optimal Programs The above derivations assume that a suitable rule set R is available ab initio, with correct qualities q i and runtimes t i , and that the deadline distribution is known. In this section, we study ways in which some of this information can be learned, and the implications of this for the bounded optimality of the resulting system. We will concentrate on learning rules and their qualities, leaving runtimes and deadline distributions for future work. The basic idea is that the learning algorithms will converge, over time, to a set of optimal components | the most accurate rules and the most accurate quality estimates for them. As this happens, the value of the agent constructed from the rules, using the quality estimates, converges to the value of l opt . T h us there are two sources of suboptimality in the learned agent: The rules in R may not be the best possible rules | they may recommend actions that are of lower utility than those that would be recommended by some other rules. There may be errors in estimating the expected utility of the rule. This can cause the algorithms given above to construct suboptimal sequences, even if the best rules are available. Our notional method for constructing bounded optimal agents 1 learns sets of individual decision procedures from episodic interactions, and 2 arranges them in a sequence using one of the algorithms described earlier so that the performance of an agent using the sequence is at least as good as that of any other such agent. We assume a parameterized learning algorithm L J ;k that will be used to learn one rule for each possible runtime k 2 f 1; : : : ; t M g. Since there is never a need to include two rules with the same runtime in the R, this obviates the need to consider the entire rule language J in the optimization process. Our setting places somewhat unusual requirements on the learning algorithm. Like most learning algorithms, L J ;k works by observing a collection T of training episodes in E, including the utility obtained for each episode. We do not, however, make a n y assumptions about the form of the correct decision rule. Instead, we make assumptions about the hypotheses, namely that they come from some nite language J k , the set of programs in J of complexity at most k. This setting has been called the agnostic learning setting by Kearns, Schapire and Sellie 1992, because no assumptions are made about the environment at all. It has been shown Theorems 4 and 5 in Kearns, Schapire and Sellie, 1992 that, for some languages J , the error in the learned approximation can be bounded to within of the best rule in J k that ts the examples, with probability 1 , . The sample size needed to guarantee these bounds is polynomial in the complexity parameter k, a s w ell as 1 and 1 . In addition to constructing the decision procedures, L J ;k outputs estimates of their quality q i . Standard Cherno -Hoe ding bounds can be used to limit the error in the quality estimate to be within q with probability 1 , q . The sample size for the estimation of quality is also polynomial in 1 q and 1 q . Thus the error in each agnostically learned rule is bounded to within of the best rule in its complexity class with probability 1 , . The error in the quality estimation of these rules is bounded by q with probability 1 , q . F rom these bounds, we can calculate a bound on the utility de cit in the agent program that we construct, in comparison to l opt : Theorem 7 Assume an architecture M J that executes sequences of decision procedures in an agnostically learnable language J whose runtimes range over 1::t M . F or real time task environments with xed time cost, xed d e adline, and stochastic deadline, we can construct a p r ogram l such that V l opt ; M ; E , V l;M;E + 2 q with probability greater than 1 , m + q , where m is the number of decision procedures in l opt . Proof: We prove this theorem for the stochastic deadline regime, where the bounded optimal program is a sequence of decision procedures. The proofs for the xed cost and xed deadline regimes, where the bounded optimal program is a singleton, follow a s a special case. Let the best decision procedures for E be the set R = fr 1 ; : : : ; r n g, and let l opt = s 1 : : : s m be an optimal sequence constructed from R . Let R = fr 1 ; : : : r n g be the set of decision procedures returned by the learning algorithm. With probability greater than 1 , m , q i , q i for all i, where q i refers to the true quality o f r i . The error in the estimated quality qi of decision procedure r i is also bounded: with probability greater than 1 , m q , jq i , q i j q for all i. Let s = s 1 : : : s m be those rules in R that come from the same runtime classes as the rules s 1 : : : s m in R . Then, by Equation  5 , we h a ve V l opt ; M ; E , V s; M; E because the error in V is a weighted average of the errors in the individual q i . Similarly, w e have j V s; M; E , V s; M; Ej q Now suppose that the sequence construction algorithm applied to R produces a sequence l = s 1 0 : : : s l 0 . By de nition, this sequence appears to be optimal according to the estimated value function V . Hence V l;M;E V s; M; E As before, we can bound the error on the estimated value: j V l;M;E , V l;M;Ej q Combining the above inequalities, we h a ve V l opt ; M ; E , V l;M;E + 2 q 2 Although the theorem has practical applications, it is mainly intended as an illustration of how a learning procedure can converge on a bounded optimal con guration. With some additional work, more general error bounds can be derived for the case in which the rule execution times t i and the real-time utility v ariation time cost, xed deadline, or deadline distribution are all estimated from the training episodes. We can also obtain error bounds for the case in which the rule language J is divided up into a smaller number of coarser runtime classes, rather than the potentially huge number that we currently use.
 Asymptotic Bounded Optimality The strict notion of bounded optimality m a y be a useful philosophical landmark from which to explore arti cial intelligence, but it may be too strong to allow many i n teresting, general results to be obtained. The same observation can be made in ordinary complexity theory: although absolute e ciency is the aim, asymptotic e ciency is the game. That a sorting algorithm is On log n rather than On 2 is considered signi cant, but replacing a multiply by 2 " b y a shift-left 1 bit" is not considered a real advance. The slack allowed by the de nitions of complexity classes is essential in building on earlier results, in obtaining robust results that are not restricted to speci c implementations, and in analysing the complexity o f algorithms that use other algorithms as subroutines. In this section, we begin by reviewing classical complexity. We then propose de nitions of asymptotic bounded optimality that have some of the same advantages, and show that classical optimality is a special case of asymptotic bounded optimality. Lastly, w e report on some preliminary investigations into the use of asymptotic bounded optimality as a theoretical tool in constructing universal real-time systems.
 Classical complexity A problem, in the classical sense, is de ned by a pair of predicates and such that output z is a solution for input x if and only if x and x; z hold. A problem instance is an input satisfying , and an algorithm for the problem class always terminates with an output z satisfying x; z given an input x satisfying x. Asymptotic complexity describes the growth rate of the worst-case runtime of an algorithm as a function of the input size. We can de ne this formally as follows. Let T a x be the runtime of algorithm a on input x, and let T a n be the maximum runtime of a on any input of size n. Then algorithm a has complexity Ofn if 9k;n 0 8n n n 0 T a n kfn Intuitively, a classically optimal algorithm is one that has the lowest possible complexity. For the purposes of constructing an asymptotic notion of bounded optimality, it will be useful to have a de nition of classical optimality that does not mention the complexity directly. This can be done as follows: De nition 11 Classically optimal algorithm: An algorithm a is classically optimal if and only if 9k;n 0 8a 0 ; n n n 0 T a n kT a 0 n To relate classical complexity to our framework, we will need to de ne the special case of task environments in which traditional programs are appropriate. In such task environments, an input is provided to the program as the initial percept, and the utility function on environment histories obeys the following constraint: De nition 12 Classical task environment: hE P ; U i is a classical task environment for problem P if V l;M;E P = uTl;M;E P if l outputs a correct solution for P 0 otherwise where Tl;M;E P is the running time for l in E P on M, M is a universal Turing machine, and u is some positive decreasing function. The notion of a problem class in classical complexity theory thus corresponds to a class of classical task environments of unbounded complexity. F or example, the Traveling Salesperson Problem contains instances with arbitrarily large numbers of cities.
 Varieties of asymptotic bounded optimality The rst thing we will need is a complexity measure on environments. Let nE be a suitable measure of the complexity o f a n e n vironment. We will assume the existence of environment classes that are of unbounded complexity. Then, by analogy with the de nition of classical optimality, w e can de ne a worst-case notion of asymptotic bounded optimality ABO. Letting V l;M;n;E be the minimum value of V l;M;E for all E in E of complexity n, we h a ve De nition 13 Worst-case asymptotic bounded optimality: an agent program l is timewise or spacewise worst-case asymptotically bounded optimal in E on M i 9k;n 0 8l 0 ; nn n 0 V l;kM;n;E V l 0 ; M ; n ; E where kM denotes a version of the machine M speeded up by a factor k or with k times more memory. In English, this means that the program is basically along the right lines if it just needs a faster larger machine to have w orst-case behaviour as good as that of any other program in all environments. If a probability distribution is associated with the environment class E, then we can use the expected value V l;M;E to de ne an average-case notion of ABO: De nition 14 Average-case asymptotic bounded optimality: an agent program l is timewise or spacewise average-case asymptotically bounded optimal in E on M i 9k 8l 0 V l;kM;E V l 0 ; M ; E For both the worst-case and average-case de nitions of ABO, we w ould be happy with a program that was ABO for a nontrivial environment on a nontrivial architecture M, unless k were enormous.  6  In the rest of the paper, we will use the worst-case de nition of ABO. Almost identical results can be obtained using the average-case de nition. The rst observation that can be made about ABO programs is that classically optimal programs are a special case of ABO programs: 7 6. The classical de nitions allow for optimality up to a constant factor k in the runtime of the algorithms. One might w onder why w e c hose to use the constant factor to expand the machine capabilities, rather than to increase the time available to the program. In the context of ordinary complexity theory, the two alternatives are exactly equivalent, but in the context of general time-dependent utilities, only the former is appropriate. It would not be possible to simply let l run k times longer," because the programs we wish to consider control their own execution time, trading it o against solution quality. One could imagine slowing down the entire environment b y a factor of k, but this is merely a less realistic version of what we propose. 7. This connection was suggested by Bart Selman. Theorem 8 A p r ogram is classically optimal for a given problem P if and only if it is timewise worst-case ABO for the corresponding classical task environment class hE P ; U i. This observation follows directly from De nitions 11, 12, and 13. In summary, the notion of ABO will provide the same degree of theoretical robustness and machine-independence for the study of bounded systems as asymptotic complexity d o e s for classical programs. Having set up a basic framework, we can now begin to exercise the de nitions.
 Universal asymptotic bounded optimality Asymptotic bounded optimality is de ned with respect to a speci c value function V. In constructing real-time systems, we w ould prefer a certain degree of independence from the temporal variation in the value function. We can achieve this by de ning a family V of value functions, di ering only in their temporal variation. By this we mean that the value function preserves the preference ordering of external actions over time, with all value functions in the family having the same preference ordering.  8  For example, in the xed-cost regime we can vary the time cost c to generate a family of value functions; in the stochastic deadline case, we can vary the deadline distribution P d to generate another family. Also, since each of the three regimes uses the same quality measure for actions, then the union of the three corresponding families is also a family. What we will show is that a single program, which w e call a universal program, can be asymptotically bounded-optimal regardless of which v alue function is chosen within any particular family. De nition 15 Universal asymptotic bounded optimality UABO: An agent program l is UABO in environment class E on M for the family of value functions V i l is ABO in E on M for every V i 2 V . A UABO program must compete with the ABO programs for every individual value function in the family. A UABO program is therefore a universal real-time solution for a given task. Do UABO programs exist? If so, how can we construct them? It turns out that we can use the scheduling construction from Russell & Zilberstein, 1991 to design UABO programs. This construction was designed to reduce task environments with unknown interrupt times to the case of known deadlines, and the same insight applies here. The construction requires the architecture M to provide program concatenation e.g., the LISP prog construct, a conditional-return construct, and the null program . The universal program l U has the form of a concatenation of individual programs of increasing runtime, with an appropriate termination test after each. It can be written as l U = l 0 l 1 l j where each l j consists of a program and a termination test. The program part in l j is any program in L M that is ABO in E for a value function V j that corresponds to a xed deadline at t d = 2 j , where is a time increment smaller than the execution time of any non-null program in L M . 8. The value function must therefore be separable  Russell & Wefald, 1989, since    Before proceeding to a statement that l U is indeed UABO, let us look at an example. Consider the simple, sequential machine architecture described earlier. Suppose we can select rules from a three-rule set with r 1 = 0 :2; 2, r 2 = 0 :5; 5 and r 3 = 0 :7; 7. Since the shortest runtime of these rules is 2 seconds, we let = 1. Then we look at the optimal programs l 0 ; l 1 ; l 2 ; l 3 ; : : : for the xed-deadline task environments with t d = 1 ; 2; 4; 8; : : : . These are: l 0 = ;l 1 = r 1 ; l 2 = r 1 ; l 3 = r 3 ; : : : Hence the sequence of programs in l U is ; r 1 ; r 1 ; r 3 ; : : : . Now consider a task environment class with a value function V i that speci es a stochastic deadline uniformly distributed over the range 0: : : 10 . For this class, l opt = r 1 r 2 is a bounded optimal sequence. 9 It turns out that l U has higher utility than l opt provided it is run on a machine that is four times faster. We can see this by plotting the two performance pro les: Q U for l U on 4M and Q opt for l opt on M. Q U dominates Q opt , a s s h o wn in Figure  6 . To establish that the l U construction yields UABO programs in general, we need to de ne a notion of worst-case performance pro le. Let Q t; l; M; n; E be the minimum value obtained by i n terrupting l at t, o ver all E in E of complexity n. W e know that each l j in l U satis es the following: 8l 0 ; n n n j V j l j ; k j M;n;E V j l 0 ; M ; n ; E for constants k j , n j . The aim is to prove that 8V i 2 V 9 k;n 0 8l 0 ; n n n 0 V i l U ; k M ; n ; E V i l 0 ; M ; n ; E Given the de nition of worst-case performance pro le, it is fairly easy to show the following lemma the proof is essentially identical to the proof of Theorem 1 in  Russell and Zilberstein, 1991:  9. Notice that, in our simple model, the output quality of a rule depends only on its execution time and not on the input complexity. This also means that worst-case and average-case behaviour are the same. Lemma 6 If l U is a universal program in E for V, and l i is ABO on M in E for V i 2 V , then Q t; l U ; k M ; n ; E dominates Q t; l i ; M ; n ; E for k 4 max j k j , n max j n j . This lemma establishes that, for a small constant penalty, w e can ignore the speci c realtime nature of the task environment in constructing bounded optimal programs. However, we still need to deal with the issue of termination. It is not possible in general for l U to terminate at an appropriate time without access to information concerning the timedependence of the utility function. For example, in a xed-time-cost task environment, the appropriate termination time depends on the value of the time cost c. For the general case with deterministic time-dependence, we can help out l U by supplying, for each V i , an aspiration level" Q i t i ; l i ; M ; n ; E, where t i is the time at which l i acts. l U terminates when it has completed an l j such that q j Q i t i ; l i ; M ; n ; E. By construction, this will happen no later than t i because of Lemma 6. Theorem 9 In task environments with deterministic time-dependence, an l U with a suitable aspiration level is UABO in E on M. With deadline heralds, the termination test is somewhat simpler and does not require any additional input to l U . Theorem 10 In a task environment with stochastic deadlines, l U is UABO in E on M if it terminates when the herald arrives. Returning to the mail-sorting example, it is fairly easy to see that l U which consists of a sequence of networks, like the optimal programs for the stochastic deadline case will be ABO in the xed-deadline regime. It is not so obvious that it is also ABO in any particular stochastic deadline case | recall that both regimes can be considered as a single family. We h a ve programmed a constructor function for universal programs, and applied it to the mail-sorter environment class. Varying the letter arrival distribution gives us di erent v alue functions V i 2 V . Figure  7  shows that l U on 4M has higher throughput and accuracy than l i opt across the entire range of arrival distributions. Given the existence of UABO programs, it is possible to consider the behaviour of compositions thereof. The simplest form of composition is functional composition, in which the output of one program is used as input by another. More complex, nested compositional structures can be entertained, including loops and conditionals  Zilberstein, 1993.  The main issue in constructing UABO compositions is how to allocate time among the components. Provided that we can solve the time allocation problem when we know the total runtime allowed, we can use the same construction technique as used above to generate composite UABO programs, where optimality is among all possible compositions of the components.  Zilberstein and Russell 1993 , show that the allocation problem can be solved in linear time in the size of the composite system, provided the composition is a tree of bounded degree.
 Conclusions And Further Work We examined three possible formal bases for arti cial intelligence, and concluded that bounded optimality provides the most appropriate goal in constructing intelligent systems. We also noted that similar notions have arisen in philosophy and game theory for more or less the same reason: the mismatch b e t ween classically optimal actions and what we h a ve called feasible behaviours|those that can be generated by an agent program running on a computing device of nite speed and size. We showed that with careful speci cation of the task environment and the computing device one can design provably bounded-optimal agents. We exhibited only very simple agents, and it is likely that bounded optimality in the strict sense is a di cult goal to achieve when a larger space of agent programs is considered. More relaxed notions such as asymptotic bounded optimality ABO may provide more theoretically robust tools for further progress. In particular, ABO promises to yield useful results on composite agent designs, allowing us to separate the problem of designing complex ABO agents into a discrete structural problem and a continuous temporal optimization problem that is tractable in many cases. Hence, we h a ve reason to be optimistic that arti cial intelligence can be usefully characterized as the study of bounded optimality. W e m a y speculate that provided the computing device is neither too small so that small changes in speed or size cause signi cant c hanges in the optimal program design nor too powerful so that classically optimal decisions can be computed feasibly, ABO designs should be stable over reasonably wide variations in machine speed and size and in environmental complexity. The details of the optimal designs may be rather arcane, and learning processes will play a large part in their discovery; we expect that the focus of this type of research will be more on questions of convergence to optimality for various structural classes than on the end result itself. Perhaps the most important implication, beyond the conceptual foundations of the eld itself, is that research on bounded optimality applies, by design, to the practice of arti cial intelligence in a way that idealized, in nite-resource models may not. We h a ve given, by way of illustrating this de nition, a bounded optimal agent: the design of a simple system consisting of sequences of decision procedures that is provably better than any other program in its class. A theorem that exhibits a bounded optimal design translates, by de nition, into an agent whose actual behaviour is desirable. There appear to be plenty o f w orthwhile directions in which to continue the exploration of bounded optimality. From a foundational point of view, one of the most interesting questions is how the concept applies to agents that can incorporate a learning component. Note that in section 5, the learning algorithm was external to the agent. In such a case, there will not necessarily be a largely stable bounded optimal con guration if the agent program is not large enough; instead, the agent will have to adapt to a shorter-term horizon and rewrite itself as it becomes obsolete. With results on the preservation of ABO under composition, we can start to examine much more interesting architectures than the simple production system studied above. For example, we can look at optimal search algorithms, where the algorithm is constrained to apply a metalevel decision procedure at each step to decide which node to expand, if any  Russell & Wefald, 1989 . We can also extend the work on asymptotic bounded optimality to provide a utility-based analogue to big-O" notation for describing the performance of agent designs, including those that are suboptimal. In the context of computational learning theory, i t i s o b vious that the stationarity requirement on the environment, which is necessary to satisfy the preconditions of PAC results, is too restrictive. The fact that the agent learns may h a ve some e ect on the distribution of future episodes, and little is known about learning in such cases  Aldous & Vazirani, 1990 . We could also relax the deterministic and episodic requirement to allow non-immediate rewards, thereby making connections to current research on reinforcement learning. The computation scheduling problem we examined is interesting in itself, and does not appear to have been studied in the operations research o r c o m binatorial optimization literature. Scheduling algorithms usually deal with physical rather than computational tasks, hence the objective function usually involves summation of outputs rather than picking the best. We w ould like to resolve the formal question of its tractability in the general case, and also to look at cases in which the solution qualities of individual processes are interdependent such as when one can use the results of another. Practical extensions include computation scheduling for parallel machines or multiple agents, and scheduling combinations of computational and physical e.g., job-shop and ow-shop processes, where objective functions are a combination of summation and maximization. The latter extension broadens the scope of applications considerably. An industrial process, such as designing and manufacturing a car, consists of both computational steps design, logistics, factory scheduling, inspection etc. and physical processes stamping, assembling, painting etc.. One can easily imagine many other applications in real-time nancial, industrial, and military contexts. It may turn out that bounded optimality is found wanting as a theoretical framework. If this is the case, we hope that it is refuted in an interesting way, so that a better framework can be created in the process.
 Appendix: Additional Proofs This appendix contains formal proofs for three subsidiary lemmata in the main body of the paper. Lemma 3 There exists an optimal sequence that is sorted in increasing order of q's. Proof: Suppose this is not the case, and s is an optimal sequence. Then there must be two adjacent rules i, i + 1 where q i q i+1 see Figure  8 . Removal of rule i + 1 yields a sequence s 0 such that Q s 0 t Q s t, from Lemma 1 and the fact that t i+2 t i+1 +t i+2 . B y Lemma 2, s 0 must also be optimal. We can repeat this removal process until s 0 is ordered by q i , proving the theorem by reductio ad absurdum.2 Lemma 4 For every sequence s = s 1 : : : s m sorted in increasing order of quality, and single step z with q z q s m , V sz V s. Proof: We calculate V sz , V s using Equation  5 and show that it is non-negative: V sz , V s = q z 1 , P d P m j=1 t j + t z , q m 1 , P d P m j=1 t j + t z = q z , q m 1 , P d P m j=1 t j + t z which is non-negative since q z q m .2 t i+1 q i+2 q i q i-1 t i q t t i+2 q i+1 Figure 8: Proof for ordering by q i ; l o wer dotted line indicates original pro le; upper dotted line indicates pro le after removal of rule i + 1 . Lemma 5 There exists an optimal sequence whose rules are in nondecreasing order of t i . Proof: Suppose this is not the case, and s is an optimal sequence. Then there must be two adjacent rules i, i + 1 where q i q i+1 and t i t i+1 see Figure  9 . Removal of rule i yields a sequence s 0 such that Q s 0 t Q s t, from Lemma 1. By Lemma 2, s 0 must also be optimal. We can repeat this removal process until s 0 is ordered by t i , proving the theorem by reductio ad absurdum.2 t i+1 q i+1 q i q i-1 t i q t t i+1 Figure  9 : Proof for ordering by t i ; dotted line indicates pro le after removal of rule i. Figure 1 : 1 Figure 1: An automated mail-sorting facility provides a simple example of an episodic, real-time task environment.
 7 = :25 A geometric intuition is given by the notion of a performance p r o le, a s s h o wn in Figure 2.
 Figure 2 : 2 Figure 2: Performance pro le for r 1 r 2 r 3 , with p d superimposed.
 Figure 3 3 Figure 3: a Accuracy pro le 1 , e ,x , for = 0 :9. b Poisson arrival distribution, for mean = 9 sec
 Figure 4 : 4 Figure 4: Graph showing the achievable utility per second as a function of the average time per letter, for the four program types. = 0 :9.
 Figure 5 : 5 Figure 5: Graphs showing the utility gain per second as a function of the arrival time variance, for the four program types for the uniform distribution with a mean of 20 seconds.
 Figure 6 : 6 Figure 6: Performance pro les for l U running on 4M, and for l opt running on M
 Figure 7 : 7 Figure 7: Throughput and accuracy improvement o f l U over l i opt , as a function of mean arrival time, = 0.2, Poisson arrivals.
 Fixed time cost: The task environment hE;Ui has a xed time cost if, for any action history pre xes A t 1 1 and A t 2 2 satisfying 1 A T 1 t 1 2 A ? and A T 2 t 2 = A T 1 t 1 2 A t 1 ,1 1 and A t 2 ,1 2 contain no action in A ? contain no action in A ? . t d ,1 2 Actions taken after t d have no e ect on utility: U E;A t 1 U E;A t 2 if U E;A t d 1 U E;A t d 2 and t t d 4.1.2 Fixed time cost Task environments with approximately xed time cost are also very common. Examples include consultations with lawyers, keeping a taxi waiting, or dithering over where to invest one's money. We can de ne a task environment with xed time cost c by comparing the utilities of actions taken at di erent times.De nition 8
			 . This usage of the term universal" derives from its use in the scheduling of randomized algorithms by Luby, Sinclair and Zuckerman 1993.
			 . For example, it is conceivable that Good and Simon really intended to refer to nding an agent design that minimizes deliberation costs in general. All their discussions, however, seem to be couched in terms of nding the right deliberation for each decision. Thus, type II or metalevel rationality coincides with bounded optimality if the bounded optimal agent is being designed for a single decision in a single situation.
			 . One would imagine that in most cases the move selected will be the same move selected by a T ype I agent, but this is in a sense accidental" because further deliberation might cause the program to abandon it.
			 Russell & Subramanian
			 . See Sackinger et al. 1992; Boser et al. 1992  for details of an actual system. The application was suggested to us by Bernhard Boser after an early presentation of our work at the 1992 NEC Symposium.