Relational Reinforcement Learning Abstract. Relational reinforcement learning is presented, a learning technique that combines reinforcement learning with relational learning or inductive logic programming. Due to the use of a more expressive representation language to represent states, actions and Q-functions, relational reinforcement learning can be potentially applied to a new range of learning tasks. One such task that we investigate is planning in the blocks world, where it is assumed that the effects of the actions are unknown to the agent and the agent has to learn a policy. Within this simple domain we show that relational reinforcement learning solves some existing problems with reinforcement learning. In particular, relational reinforcement learning allows us to employ structural representations, to abstract from specific goals pursued and to exploit the results of previous learning phases when addressing new (more complex) situations. Within the field of machine learning, both reinforcement learning (Kaelbling et al., 1996) and inductive logic programming (or relational learning) (Muggleton & De Raedt, 1994; Lavrač & Džeroski, 1994) have received a lot of attention since the early nineties. It is therefore no surprise that both Leslie Pack Kaelbling and Richard Sutton (in their invited talks at IJCAI-97, Nagoya, Japan) suggested to study the combination of these two fields.. From the reinforcement learning point of view, this could significantly extend the application perspective. Most representations used in reinforcement learning are inadequate for describing planning tasks such as the simple blocks world. Even reinforcement learning work that involves generalization has largely employed an attribute-value representation. Due to the use of variables in relational representations, it is possible to abstract from specific details of the learning tasks, such as the specific goal pursued. Indeed, when learning to plan in the blocks world, one would expect that the results of learning how to stack block a onto block b would be similar to stacking c onto d. Current approaches to reinforcement learning have to retrain from scratch if the goal is changed in this manner, while for relational reinforcement learning such retraining is unnecessary. Relational reinforcement learning also allows us to exploit and apply the results of learning in a simple domain when learning in a more complex domain (e.g., going from 3 blocks to 4 blocks in the blocks world).. From the inductive logic programming point of view, it is important to address domains such as reinforcement learning. So far, inductive logic programming has mainly studied concept-learning, and largely ignored the rest of machine learning. By demonstrating the potential of relational representations for reinforcement learning, we hope to show that the relational learning methodology does not only apply to concept-learning but to the whole field of machine learning.. With this in mind, we present an approach to relational reinforcement learning and apply it to simple planning tasks in the blocks world. The planning task involves learning a policy to select actions. Learning is necessary as the planning agent does not know the effects of its actions. Relational reinforcement learning employs the Q-learning method (Watkins & Dayan, 1992; Kaelbling et al., 1996; Mitchell, 1997) where the Q-function is learned using a relational regression tree algorithm (see Kramer, 1996) ). A state is represented relationally as a set of ground facts. A relational regression tree in this context takes as input a relational description of a state, a goal and an action, and produces the corresponding Q-value. The Q-learning method can also be adapted in order to learn the P-function, an explicit representation of the policy implicitly represented by the Q-function. The P-function, which is represented as a first order logical decision tree, takes as input a state, an action, and a goal and predicts whether the action is optimal or not.. The paper is organized as follows. In Section 2, we view planning (under uncertainty) as a reinforcement learning task. In Section 3, we briefly review Q-learning and show how Q-learning can be used to learn a P-function. Section 4 briefly reviews decision trees while focusing on logical decision trees. Section 5 introduces relational reinforcement learning that combines Q-learning and logical regression trees, as well as P-learning and logical decision trees. Section 6 presents a variety of experiments aimed at exploring the potential of relational reinforcement learning. Section 7 concludes, touches upon related work and discusses avenues for further work. We have presented an approach to planning with incomplete knowledge that combines reinforcement learning and relational learning into a technique called relational reinforcement learning. The advantages of this approach include the ability to use structured representations, which enables us to also describe infinite worlds, and the ability to use variables, which allows us to abstract away from specific details of the situations (such as, e.g., the goal, the number of blocks). The ability to carry over the policies learned in simple situations (with few blocks) to more complex situations was demonstrated. It is hard to see how this could be realized without the use of relational representations. We continue the discussion by discussing scalability, related work and further work. The main contribution of our work is to address the generalization problem in reinforcement learning within a relational setting. The task of finding optimal plans within the blocks world was already considered by Langley in his book (Langley, 1996) , to illustrate reinforcement learning. However, instead of using a relational learner for generalization he employs a neural net using a fixed set of propositional features. Indeed, typical generalizers in reinforcement learning are based on neural nets, cf. e.g. (Tesauro, 1991) , whereas we employ decision trees.. The use of decision trees in a reinforcement context is not new. It was first proposed by Chapman's and Kaelbling's (Chapman & Kaelbling, 1991) , who developed an incremental decision tree learning algorithm with special heuristics to cope with concept-drift (Widmer & Kubat, 1998) in the reinforcement learning context. Our approach is distinguished from the one by Chapman and Kaelbling by the use of a relational representation. However, it would be extremely useful to integrate algorithm by Chapman and Kaelbling (i.e . the heuristics and the incremental aspect) with the representations provided by TILDE and TILDE-RT.. Another piece of work that is very much related to our presentation of RRL is that of Baum (Baum, 1996) . He uses a kind of genetic approach to learning rules in the blocks world for goals such as stacking and unstacking. To this aim he employs a special rule language (at a level between propositional and first order logic) for expressing policies and uses genetic ingredients to learn and modify the set of rules. Whereas this approach is elegant, it does not employ the basic principles of temporal difference learning as we do.. The combination of learning and planning has received a lot of attention in the artificial intelligence literature (see (Langley, 1996) for an excellent overview). Also, most approaches to learning in a planning context do employ relational representations. It can be no surprise that various types of learning tasks have been considered in this context.. -A first line of research attempts to improve the domain knowledge of the planner. This corresponds to learning more accurate definitions of the operators, i.e. the effects, preand post-conditions. This approach has been integrated in Prodigy, cf. (Carbonell & Gill, 1990) . The planner then exploits the learned knowledge in order to construct better plans. The difference with relational reinforcement learning is that our approach does not rely on a planner. This is important as one might consider relational reinforcement learning outside a planning context. -A second line of research concerns the learning of control knowledge. E.g. the work on LEX (Mitchell et al., l983) and SAGE (Langley, 1985) learned when to apply certain operators. The goal of this work is thus similar to that of relational reinforcement learning. However, the approach is quite different. E.g. to solve symbolic integration problems, LEX would construct a search tree (a trace) in which all legal operators were applied to a given integration problem until a solution was found at a certain depth. Once the solution was found, LEX would label all applications of operators on the path leading to the (optimal) solution as positive examples, and all applications diverging from this path as negative examples. The examples were then fed into a kind of relational learning algorithm and used to refine the control knowledge. So, the mechanism for learning is quite different. The LEX and SAGE method does not apply in the context of autonomous agents, because it assumes that one can backtrack to earlier states (which may not be possible-and which is certainly problematic without adequate domain knowledge). On the other hand, our method to construct examples for learning the P-tree is certainly similar in spirit to LEX. -A third line of research analytically learns control knowledge often using a form of explanation based learning. The difference with relational reinforcement learning is that explanation based learning relies on complete knowledge about the domain.. The work by Stone and Veloso (Stone & Veloso, 1999) is closely related to ours in two ways. First, they use decision-trees to learn a Q-function, thereby generalizing. Second, they use a mapping on states to transform large state-spaces into learnable ones. This mapping is hand-coded. Our approach uses a mapping which generalizes across state-action pairs; this mapping is implicitly defined by the relational representation and the background knowledge.. Somewhat related to our approach is work on hierarchical reinforcement learning, such as options (Sutton et al., 2000) . Options are macro actions defined by a region of the state space where execution can begin, a policy and a termination condition. Options can be viewed as background knowledge, albeit different in nature from the one currently used by RRL. We performed two different sets of experiments. In the first set of experiments, the policies were learned from state spaces in which the number of blocks was held constant, cf. Section 6.3. In the second set of experiments, discussed in Section 6.4, we varied the number of blocks while learning.. In both sets of experiments we tried out different goals such as stacking, unstacking and on(a, b) (cf. Section 6.2.1 for a discussion on the goals pursued) and used the background knowledge and parameter settings discussed in Section 6.2.2, except for the experiments described in Section 6.6. Prolog code specifying the optimal policies for achieving these goals is given in Table 7 . The optimal policy for unstacking is the simplest: moving any block (that is not already on the floor) to the floor is optimal. The policy for stacking is a bit more complex: moving a block to the highest stack is optimal. The policy for achieving on(a, b) is the most complex: if possible, a should be moved to b; otherwise, a block above a or b should be moved away (but not to the stack where b or a are).. The above ordering of the three goals also corresponds to the number of reachable goal states, in decreasing order. A reachable goal state is a goal state (where the goal is satisfied) and which can be reached from a non-goal state in a single step (by applying one action). For the unstack goal and a fixed number of blocks there is only a single state that satisfies the goal. For the stack goal, given n blocks there are n! goal states. The number of possible states increases exponentially with the number of blocks. This is summarized in Table 8 .. One point that should be clear from this table is that for some of the goals (e.g., unstacking with 10 blocks) the reinforcement learning algorithm described in Section 3.2 is inapplicable: the probability of reaching the goal state by random exploration is extremely low (given only 1 goal state in 58 941 091). Another point that this table demonstrates is the difficulty of learning policies in the blocks world. Despite the fact that the blocks world is an artificial toy domain, policy learning can become very complex due to the large number of possible states.. Note also that for unstack, the number of possible actions increases as one gets closer to the goal states: one step away from the goal state there are (n − 1)(n − 2) + 1 possible actions. For stack, on the other hand, there are only two possible actions if we are one step away from a goal state.

O B I O O B I O O O O O O O B I O O O O B I I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O B I O O B I I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Genetic Stability and Territorial Structure Facilitate the Evolution of Tag-Mediated Altruism Abstract Evolutionary theorists have long been interested in the conditions that permit the evolution of altruistic cooperation. Recent work has demonstrated that altruistic donation can evolve in surprisingly simple models, in which agents base their decisions to donate solely on the similarity of evolved ''tags'' relative to evolved tag-difference tolerances. There is disagreement, however, about the conditions under which tag-mediated altruism will in fact evolve. Here we vary two critical parameters in a standard model of tag-mediated altruism -genetic stability and territorial structureand show that altruism evolves in a wide range of conditions. We demonstrate the evolution of significant levels of altruism even when the immediate costs to donors equal the benefits to recipients. We describe the mechanism that permits the emergence of altruism in the model as a form of kin selection that is facilitated by interactions between altruism, genetic drift, and fecundity.Evolutionary theorists have long been interested in the conditions under which altruism can be produced by evolution. Prior work has explained the evolution of altruistic behavior toward kin [6, 7] , reciprocating partners [1, 20] , and partners with good reputations [14] , as well as non-reciprocal altruism in Mendelian populations driven by population structure [22] . Recently attempts have been made to explain altruism that is based solely on partner similarity, as assessed relative to arbitrary tags [9] that can serve as models of either genetic or cultural markers [16 -18]. These attempts, if successful, would provide extremely general explanations, as they make few assumptions regarding genetic structure or the knowledge and abilities of the evolving agents. In particular, they would help to explain how altruism can evolve among agents that have no explicit knowledge of kinship, past behavior, or reputation. The models would also be applicable to organisms that are far too simple to engage in activities that underlie some other models of the evolution of altruism, such as mate selection [11] and explicit repression of competition [4] .There is disagreement, however, about the conditions under which tag-mediated altruism will in fact evolve. Roberts and Sherratt [18] argue that altruism is actually ''built in'' to the model of Riolo et al. [16], and that altruism no longer evolves if an implicit bias favoring cooperation among agents with identical tags is removed. We show here, however, that tag-mediated altruism does in fact evolve in a range of conditions, even in the context of Roberts and Sherratt's modifications. In particular, we show that it evolves when lower mutation rates are used and when territorial structure limits the interaction radius of each agent. We also show that this holds even when the immediate costs and benefits of altruistic acts are equivalent. We discuss this surprising result and the mechanisms that produce it in the final section of the article.We began with a computational model equivalent to those used in the prior work [16] , in which a population of 100 agents is progressed through a number of generations (30,000 in prior work, 400,000 here). Each agent has a tag s and a tolerance T, both of which are represented as floating point numbers and uniformly sampled from [0, 1] in the initial generation. All agents begin each generation with a score of 0 and are then paired with P=3 randomly selected others for potential acts of altruism. If the difference between the tags of the two agents is less than or equal to the potential donor's tolerance, then the altruistic act (donation) is performed: a cost c is deducted from the donor's score, and a benefit b is added to the recipient's score. The resulting scores may be positive, negative, or zero. For most of our runs we used c=0.1, b=1.0, as in most of the prior work, but we also studied other values of c and b. After all donation attempts are completed, the population of the next generation is produced by pairing each agent a i in a reproductive tournament with another randomly selected agent a j and producing, at the location of a i , a child from the parent with the higher score (or from a i in case of a tie). 1 The child inherits its tag and tolerance from the parent, but each may each be mutated with probability m (= 0.1 in the prior work, though we vary it here). When a tag is mutated, it is replaced with a new tag randomly selected from [0, 1] ; when a tolerance is mutated, it is summed with mean 0, standard deviation 0.01 Gaussian noise.2 This mutation scheme, like the donation, selection, and reproduction schemes described above, is somewhat arbitrary, but it is also standard in the literature.The primary measure of interest is the average donation rate, which ranged from 2.1% to 79.2% in the data presented in the original report by Riolo et al. [16] . We also measured tag diversity by counting the number of unique tags in each generation and averaging this number over all generations in a run. Riolo et al. did not allow tolerances to mutate below 0, so agents with identical tags would always donate to one another. Roberts and Sherratt [18] reported that the removal of this restriction (implemented by changing the lower bound for T from 0 to À10 À6 ) prevents the emergence of altruism; they observed donation rates of 1.48%, as opposed to Riolo et al.'s 73.6%. Riolo et al. replied that the two models have opposite, strong biases regarding cooperation among agents with identical tags, and that cooperation continues to evolve in models with intermediate biases [17] . Here we investigated only the condition that prevented the evolution of cooperation for Roberts and Sherratt (T z À10 À6 ), but we varied other parameters and found that cooperation does in fact evolve in a range of conditions.The mutation rate used in the prior work, 0.1, is exceptionally high compared to naturally occurring rates. We ran experiments with additional mutation rates m (0.001, 0.01, and 0.5) and found that in more stable genetic environments (with lower m) higher donation rates often evolve; for example, using Roberts and Sherratt's parameters and varying the mutation rate, we observed a donation rate of 37.61% for m=0.001. Tag diversity was lower in this condition than in Roberts and Sherratt's condition, but it was still considerable; we saw an average of 33.17 unique tags in each generation. Our results agreed well with those of Roberts and Sherratt for m=0.1 (1.30% donation, tag diversity 3 A variety of other diversity measures might also be informative, including measures of ''effective diversity'' that are sensitive to the tolerance levels found in a population.2 The lower bound for the resulting tolerance is discussed below; there is no upper bound.1 The location of the child is not specified in the prior literature; it becomes relevant only under the modifications presented here.Evolution of Tag-Mediated Altruism L. Spector and J. Klein 79.87), and we also found low donation rates with m=0.5 (2.59%) and m=0.01 (3.93%). These results are graphed in Figure 1 , which also displays the results of additional experiments discussed below. Natural populations also have territorial structure that limits both the potential recipients of altruistic acts and the reproductive competitors; in some cases such a structure has been shown to affect the evolution of altruism. For example, Nowak and May showed that a regular twodimensional grid structure facilitates the evolution of cooperation in a prisoner's dilemma game [13] . Even when populations are large and irregularly structured, patterns of interaction tend to form ''small world networks'' that have been shown to affect game theoretic dynamics [21] , and several researchers have characterized relations between population structure and evolutionary dynamics more generally [10, 22] . An earlier analytic model of Eshel [3] predicted, in agreement with Wright [24] , that low mobility in a population with territorial structure would produce a ''neighbor effect'' that promotes the evolution of altruism. Others have subsequently studied the effects of low mobility (a.k.a. ''viscosity''), which can also sometimes inhibit altruism by intensifying competition among neighboring altruists [12, 23] . For example, recent work by Hauert and Doebeli demonstrates that spatial structure can inhibit the evolution of cooperation in the ''snowdrift game '' [8]. Riolo et al. described possible connections between territorial structure and the evolution of altruism in their model, but without presenting data [17] . More recently Axelrod et al.[2] demonstrated the emergence of tag-mediated altruism in populations structured as two-dimensional lattices.Here we have modeled a simple form of territorial structure by considering the population as a one-dimensional array wrapped around a ring. Each agent's interactions are limited to others within radius R; for example, with R=5 each agent has 10 potential partners for altruistic acts and for reproductive competition (5 on each side). The model of Roberts and Sherratt results from setting R=50 (as does the model of Riolo et al. [16] , with the additional setting of T z0). Note that this scheme, while simple, has features more similar to those of natural populations than some of the alternatives that have been explored in the literature; for example, in contrast to the scheme used by Hauert and Doebeli [8] , agents in our model may interact with agents that are not directly adjacent, and the amount of overlap between neighborhoods decreases relatively smoothly for pairs of agents at successively greater distances from one another. A more standard cellular-automaton-like model, in which agents can interact only when immediately adjacent, results from setting R=1.We observed that as R decreases, the amount of altruism often increases. For example, in Roberts and Sherratt's condition (aside from this change) we observed donation rates of 12.48% for R=10 (tag diversity 58.74), 31.74% for R=5 (tag diversity 40.66), and 55.80% for R=1 (tag diversity 41.81). Notice that the condition R=1 produces both more altruism and more diversity than R=5; as discussed below, this demonstrates that the observed altruism is not a simple product of homogeneity. Figure 1 . Donation rate as a function of mutation rate and interaction radius, with a cost/benefit ratio c/b=0.1 (as in prior work). The configuration described by Roberts and Sherratt [18] is marked with the circle and ''R&S.'' Here 92 independent runs were conducted for each of 72 conditions. A log scale was used for the interaction radius axis to highlight the behavior of the system at small radii.Artificial Life Volume 12, Number 4 555Evolution of Tag-Mediated Altruism L. Spector and J. Klein Territorial structure can be combined with lower mutation rates, producing cumulative effects. For example, R=1 with m=0.01 yields a donation rate of 84.11%, while the change to R alone yields only 55.80% and the change to m alone yields only 3.93%. By varying both R and m we produce the landscape of altruism shown in Figure 1 . Donation rates of up to 84.11% (with tag diversity 9.52) were obtained by combining changes in mutation rate with changes in territorial structure. We conclude that while the conditions studied by Roberts and Sherratt produce minimal altruism, significant altruism evolves in many other reasonable conditions.Altruism can evolve even when the cost/benefit ratio of altruistic donation is much higher than the value 0.1 considered above. The value 0.1 was used in prior work without substantial justification, but it might be considered to model alarm calls, or the sharing of surplus food, or other altruistic acts that benefit the recipient greatly at little cost to the actor. These justifications notwithstanding, one might wonder about the extent to which the observed effects depend on the large net gain to the population, and about the ability of the model to explain the evolution of more costly forms of altruism.Riolo et al. found that donation rates fell to 24.7% when c/b reached 0.5 (even though, in their model, Tz0). In contrast, we observed a donation rate of 75.82% with c/b=0.5, m=0.01, and R=1. At higher ratios c/b one would expect less altruism, and Hamilton's rule (discussed further below) might lead one to expect altruism to approach zero with c/b=1.0. Consistent with this expectation, Riolo et al. report that when c/b reaches 0.6 the donation rate drops to 2.2%. Surprisingly, however, we find that donation rates average 12.13% even with c/b = 1.0 (with m=0.01 and R=1). This effect is remarkably consistent; although the rate fluctuates over the course of a run, in our 92 runs in the condition m=0.01, R=1, c/b = 1.0 we found average donation rates ranging only from 11.19% to 13.01%. This range is an order of magnitude higher than rates found in many of our other conditions and much higher than control runs in which large penalties were assigned to all altruists (producing rates of less than 0.6%). This means that nontrivial levels of altruism can arise even when donation produces no net gain to the population, a situation analogous to the sharing of scarce food in a natural ecosystem. The levels of donation found in our simulations with c/b = 1.0 are graphed in Figure 2 . The effect is the strongest for R=1, a condition that resembles the cellular automata upon which many other spatial models have been based; but there are also effects at other values of R. We discuss the mechanism responsible for this effect in the following section. The simulation results have several implications: they provide evidence supporting the viability of tag-mediated models of altruism in general, they demonstrate the effects of genetic stability and territorial structure on these models, and they demonstrate that altruism can arise even when the immediate cost to the donor is equivalent to the immediate benefit to the recipient. We address these implications in this order.The data show that alternative but reasonable mutation rates and a simple form of territorial structure both facilitate the evolution of altruism in a standard tag-mediated model, potentially explaining prior non-altruistic outcomes as resulting from particular assumptions about these parameters. Consistent with discussions of these factors in the literature, our diversity data suggests that altruism in the model depends to some extent on homogeneity among interacting agents. It should be noted, however, that we observed significant levels of diversity in all conditions and that we observed no cases in which diversity collapsed completely. In all of our conditions the average number of unique tags in a generation was at least 4, and in many of our conditions with high levels of altruism the average was over 40; the diversity averaged over all of the runs graphed in Figures 1 and 2 was 67.09. Considering that the population size is just 100, these are remarkably diverse populations. The observed altruism is therefore not a simple consequence of complete homogeneity.Neither is the observed altruism a simple consequence of mutation rates that are so low as to prevent any occurrence of negative-tolerance defectors. We observed substantial altruism with m=0.01, which produces about one mutation every generation, and a mutation to the agent at any particular location about once every 100 generations. The initial population is likely to include at least one agent with tolerance around 0.01, and there is a strong initial selective advantage to low tolerance. Since, in addition, tolerance mutation is performed by summation with Gaussian noise of mean 0 and standard deviation 0.01, one would expect to produce at least one individual with zero or negative tolerance within the first few generations. Since we ran our simulations for 400,000 generations, there was ample time for low or negative tolerances to arise through mutations and to spread through the population. One would expect the average tolerance to converge to values near zero quickly, and for most of the tolerance values to cluster around zero even when our low mutation rates are in effect. This is indeed what we observe; the average tolerance, averaged over all of our conditions, was just 0.014. Mutations to zero or negative tolerance were therefore quite common.It is also important to note that the relatively high incidence of altruistic behavior in these simulations cannot be attributed to random variation alone. Mutation does constantly introduce small numbers of new altruists, an effect that increases with higher mutation rates. Figures 1 and 2 show, however, that higher mutation rates often lead to lower levels of altruism in the model discussed here. In these cases mutation is disrupting heredity and thereby dampening the effects of selection. Additional tests with the extreme mutation rate of m=1.0 in the condition c=b=1 produced an average donation rate of 4.88, which is less than half of that produced with m=0.01.The altruism-promoting effects of lower mutation rates and territorial structure can be seen as arising from a dynamic similar to Eshel's ''neighbor effect '' [3]. Altruistic agents are always vulnerable, but they are less likely to meet defectors in environments with low mutation rates and/or small interaction radii. Although there is no direct reciprocation, no explicit knowledge of kinship, and no knowledge of past behavior or reputation in any of the models under consideration here, altruistic agents in sufficiently stable environments will generally find themselves in the vicinity of other altruistic agents, many of which will in fact be kin. Tag-mediated altruism in such models can therefore be seen as a form of probabilistic kin recognition, in which agents with similar tags are not explicitly recognized as kin but have a high probability of being kin nevertheless.The fact that we observe significant altruism even when c=b=1 is surprising and might even be thought to violate Hamilton's rule, which states that altruism can evolve only in environments in which the cost to the altruistic donor is less than the product of a relatedness coefficient times the benefit to the recipient; that is, c < rb, where r V 1 [6, 7] . Whether or not the results reported here constitute a violation depends on the interpretation of c and b (which are variously construed as immediate costs Artificial Life Volume 12, Number 4 557Evolution of Tag-Mediated Altruism L. Spector and J. Klein and benefits to individuals, as changes to expected numbers of offspring, and as contributions to ''inclusive fitness'') and also on the interpretation of r (about which there is also debate; see [15] ). The literature on the proper interpretation of Hamilton's rule is rich (see, for example, [5] ), and the significance of the results reported here in the context of this literature is an interesting, open question.We investigated the mechanism responsible for the evolution of altruism in the condition c=b=1 by observing graphic depictions of simulations over evolutionary time, displaying individual tags, tolerances, and scores using grayscale and color values. From these observations we derived simple illustrations of some of the processes by which altruism can form and grow, even when c/b=1. The simplest illustration begins with a population with R=1 in which all agents have negative tolerances; that is, in which all agents are initially selfish (Figure 3a) . 4 Here there is no selection pressure on tag values, and it is possible for tag drift to produce high levels of tag diversity (Figure 3b) , even if the original tag values were all identical. When tags are sufficiently diverse, then tolerances can themselves begin to drift, as small positive tolerances will have no effect (Figure 3c ). This twostage genetic drift process explains how the first potentially altruistic (positive tolerance) agents can survive to produce offspring.Next consider two adjacent agents with positive tolerance, one of which does actually produce a net positive donation to the other ( Figure 3d) ; it may do this, for example, because its tolerance has mutated to a larger value than its differently tagged neighbor, or because the two agents are clones of the same parent and their donations to one another are asymmetric by chance. After this happens, the true altruist will not reproduce, but the recipient, which is also a potential altruist (it has a positive tolerance), is likely to be unusually fecund. It will win reproductive tournaments against the loser, and it will also win most tournaments against others, since it has received donations that raise its score above that of agents in a selfish population. It is therefore likely to produce two or more adjacent children which, unless the mutation rate is unusually high, are likely to be clones of one another (Figure 3e ). Because these children have identical tags and non-negative tolerances, they will also donate to one another. If they donate symmetrically, then they are neither advantaged nor disadvantaged by the donations, but if they donate asymmetrically, then the winners are again likely to be unusually fecund. When a winner is at the edge of the block of altruists, it will also be likely to win tournaments against differently tagged neighbors, the scores of which will be those of agents in a selfish population. This interaction may cause the block of altruists to grow, as the agents near the ends are sacrificed to allow the agents at the ends to extend the altruistic community ( Figure 3f ) . As a result, altruism-supporting genes and actual acts of altruism may increase in the population until equilibrium is reached with the trend toward selfishness produced by defections. The fact that this equilibrium need not be near zero is a consequence of the way in which genetic drift, geography, altruism, and fecundity interact in the model. Ironically, it is the disparities that arise among the altruistic agents because of random donation asymmetries that allow some of them to gain the upper hand on their selfish neighbors, and thereby to propagate altruistic behavior to a wider community.With reasonable changes to another critical parameter, reproductive tournament group size, this mechanism can support even more extreme forms of altruism in which the cost actually exceeds the benefit. While our preliminary experiments with c/b > 1 produced no significant altruism, we have observed consistently high levels of altruism when reproductive tournaments involve more than two agents. For example, with c=1.1, b=1, R=5, and a tournament size of 4 (meaning that the agent and 3 others in the neighborhood of 10 agents participate in each tournament), we observe donation rates consistently over 30%. The appropriate setting for the tournament size parameter is not obvious, and it seems reasonable to assume that the appropriate setting will vary for different ecological circumstances. For some settings of this parameter the mechanism described in this article can explain the evolution of costly forms of altruism.In related work we have demonstrated the emergence of significant levels of altruism with c=b=1 in more complex models involving continuous 3D dynamics and agents controlled by evolved 4 In our actual simulations the tags and tolerances in the initial population are randomly assigned, as discussed above. We discuss a purely selfish population here only for the sake of illustration; this presents the most challenging case for the emergence of altruism.Evolution of Tag-Mediated Altruism L. Spector and J. Klein computer programs [19] . These models are difficult to analyze, and we cannot yet be certain that the effects described in this article are responsible for the results. Nonetheless, the evolution of tagmediated altruism in complex models, even with c=b=1, suggests that tag-mediated altruism can evolve in a variety of interesting circumstances.Acknowledgments The simulation results have several implications: they provide evidence supporting the viability of tag-mediated models of altruism in general, they demonstrate the effects of genetic stability and territorial structure on these models, and they demonstrate that altruism can arise even when the immediate cost to the donor is equivalent to the immediate benefit to the recipient. We address these implications in this order.. The data show that alternative but reasonable mutation rates and a simple form of territorial structure both facilitate the evolution of altruism in a standard tag-mediated model, potentially explaining prior non-altruistic outcomes as resulting from particular assumptions about these parameters. Consistent with discussions of these factors in the literature, our diversity data suggests that altruism in the model depends to some extent on homogeneity among interacting agents. It should be noted, however, that we observed significant levels of diversity in all conditions and that we observed no cases in which diversity collapsed completely. In all of our conditions the average number of unique tags in a generation was at least 4, and in many of our conditions with high levels of altruism the average was over 40; the diversity averaged over all of the runs graphed in Figures 1 and 2 was 67.09. Considering that the population size is just 100, these are remarkably diverse populations. The observed altruism is therefore not a simple consequence of complete homogeneity.. Neither is the observed altruism a simple consequence of mutation rates that are so low as to prevent any occurrence of negative-tolerance defectors. We observed substantial altruism with m=0.01, which produces about one mutation every generation, and a mutation to the agent at any particular location about once every 100 generations. The initial population is likely to include at least one agent with tolerance around 0.01, and there is a strong initial selective advantage to low tolerance. Since, in addition, tolerance mutation is performed by summation with Gaussian noise of mean 0 and standard deviation 0.01, one would expect to produce at least one individual with zero or negative tolerance within the first few generations. Since we ran our simulations for 400,000 generations, there was ample time for low or negative tolerances to arise through mutations and to spread through the population. One would expect the average tolerance to converge to values near zero quickly, and for most of the tolerance values to cluster around zero even when our low mutation rates are in effect. This is indeed what we observe; the average tolerance, averaged over all of our conditions, was just 0.014. Mutations to zero or negative tolerance were therefore quite common.. It is also important to note that the relatively high incidence of altruistic behavior in these simulations cannot be attributed to random variation alone. Mutation does constantly introduce small numbers of new altruists, an effect that increases with higher mutation rates. Figures 1 and 2 show, however, that higher mutation rates often lead to lower levels of altruism in the model discussed here. In these cases mutation is disrupting heredity and thereby dampening the effects of selection. Additional tests with the extreme mutation rate of m=1.0 in the condition c=b=1 produced an average donation rate of 4.88, which is less than half of that produced with m=0.01.. The altruism-promoting effects of lower mutation rates and territorial structure can be seen as arising from a dynamic similar to Eshel's ''neighbor effect'' [3] . Altruistic agents are always vulnerable, but they are less likely to meet defectors in environments with low mutation rates and/or small interaction radii. Although there is no direct reciprocation, no explicit knowledge of kinship, and no knowledge of past behavior or reputation in any of the models under consideration here, altruistic agents in sufficiently stable environments will generally find themselves in the vicinity of other altruistic agents, many of which will in fact be kin. Tag-mediated altruism in such models can therefore be seen as a form of probabilistic kin recognition, in which agents with similar tags are not explicitly recognized as kin but have a high probability of being kin nevertheless.. The fact that we observe significant altruism even when c=b=1 is surprising and might even be thought to violate Hamilton's rule, which states that altruism can evolve only in environments in which the cost to the altruistic donor is less than the product of a relatedness coefficient times the benefit to the recipient; that is, c < rb, where r V 1 [6, 7] . Whether or not the results reported here constitute a violation depends on the interpretation of c and b (which are variously construed as immediate costs and benefits to individuals, as changes to expected numbers of offspring, and as contributions to ''inclusive fitness'') and also on the interpretation of r (about which there is also debate; see [15] ). The literature on the proper interpretation of Hamilton's rule is rich (see, for example, [5] ), and the significance of the results reported here in the context of this literature is an interesting, open question.. We investigated the mechanism responsible for the evolution of altruism in the condition c=b=1 by observing graphic depictions of simulations over evolutionary time, displaying individual tags, tolerances, and scores using grayscale and color values. From these observations we derived simple illustrations of some of the processes by which altruism can form and grow, even when c/b=1. The simplest illustration begins with a population with R=1 in which all agents have negative tolerances; that is, in which all agents are initially selfish (Figure 3a) . 4 Here there is no selection pressure on tag values, and it is possible for tag drift to produce high levels of tag diversity (Figure 3b) , even if the original tag values were all identical. When tags are sufficiently diverse, then tolerances can themselves begin to drift, as small positive tolerances will have no effect (Figure 3c ). This twostage genetic drift process explains how the first potentially altruistic (positive tolerance) agents can survive to produce offspring.. Next consider two adjacent agents with positive tolerance, one of which does actually produce a net positive donation to the other ( Figure 3d) ; it may do this, for example, because its tolerance has mutated to a larger value than its differently tagged neighbor, or because the two agents are clones of the same parent and their donations to one another are asymmetric by chance. After this happens, the true altruist will not reproduce, but the recipient, which is also a potential altruist (it has a positive tolerance), is likely to be unusually fecund. It will win reproductive tournaments against the loser, and it will also win most tournaments against others, since it has received donations that raise its score above that of agents in a selfish population. It is therefore likely to produce two or more adjacent children which, unless the mutation rate is unusually high, are likely to be clones of one another (Figure 3e ). Because these children have identical tags and non-negative tolerances, they will also donate to one another. If they donate symmetrically, then they are neither advantaged nor disadvantaged by the donations, but if they donate asymmetrically, then the winners are again likely to be unusually fecund. When a winner is at the edge of the block of altruists, it will also be likely to win tournaments against differently tagged neighbors, the scores of which will be those of agents in a selfish population. This interaction may cause the block of altruists to grow, as the agents near the ends are sacrificed to allow the agents at the ends to extend the altruistic community ( Figure 3f ) . As a result, altruism-supporting genes and actual acts of altruism may increase in the population until equilibrium is reached with the trend toward selfishness produced by defections. The fact that this equilibrium need not be near zero is a consequence of the way in which genetic drift, geography, altruism, and fecundity interact in the model. Ironically, it is the disparities that arise among the altruistic agents because of random donation asymmetries that allow some of them to gain the upper hand on their selfish neighbors, and thereby to propagate altruistic behavior to a wider community.. With reasonable changes to another critical parameter, reproductive tournament group size, this mechanism can support even more extreme forms of altruism in which the cost actually exceeds the benefit. While our preliminary experiments with c/b > 1 produced no significant altruism, we have observed consistently high levels of altruism when reproductive tournaments involve more than two agents. For example, with c=1.1, b=1, R=5, and a tournament size of 4 (meaning that the agent and 3 others in the neighborhood of 10 agents participate in each tournament), we observe donation rates consistently over 30%. The appropriate setting for the tournament size parameter is not obvious, and it seems reasonable to assume that the appropriate setting will vary for different ecological circumstances. For some settings of this parameter the mechanism described in this article can explain the evolution of costly forms of altruism.. In related work we have demonstrated the emergence of significant levels of altruism with c=b=1 in more complex models involving continuous 3D dynamics and agents controlled by evolved computer programs [19] . These models are difficult to analyze, and we cannot yet be certain that the effects described in this article are responsible for the results. Nonetheless, the evolution of tagmediated altruism in complex models, even with c=b=1, suggests that tag-mediated altruism can evolve in a variety of interesting circumstances. (c) In neighborhoods with high tag diversity, small positive tolerances will have no effect and tolerances will drift. (d) When a positive-tolerance agent (shown with the heavy outline) receives net positive donations, either from a neighbor with higher tolerance or from a clone with which interactions were asymmetric by chance, the agent will be likely to sire adjacent clones in the subsequent generation. (e) If the adjacent positive-tolerance clones donate to one another symmetrically, then they will be neither advantaged nor disadvantaged by their potential altruism, but if they donate asymmetrically (which is likely), then some of them will fail to reproduce, but others may again be unusually fecund. (f ) This process can repeat, causing the block of altruists to grow as agents at the ends use donations from agents within the block to outcompete other neighbors. Eventually a defector will arise (through a mutation to negative tolerance), and the block of altruists will degrade from within.

B I O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B B I O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Web-based information content and its application to concept-based video retrieval Semantic similarity between words or phrases is frequently used to find matching correlations between search queries and documents when straightforward matching of terms fails. This is particularly important for searching in visual databases, where pictures or video clips have been automatically tagged with a small set of semantic concepts based on analysis and classification of the visual content. Here, the textual description of documents is very limited, and semantic similarity based on WordNet's cognitive synonym structure, along with information content derived from term frequencies, can help to bridge the gap between an arbitrary textual query and a limited vocabulary of visual concepts. This approach, termed concept-based retrieval, has received significant attention over the last few years, and its success is highly dependent on the quality of the similarity measure used to map textual query terms to visual concepts.In this paper, we consider some issues of semantic similarity measures based on Information Content (IC), and propose a way to improve them. In particular, we note that most IC-based similarity measures are derived from a small and relatively outdated corpus (the Brown corpus), which does not adequately capture the usage pattern of many contemporary terms: for example, out of more than 150,000 WordNet terms, only about 36,000 are represented. This shortcoming reflects very negatively on the coverage of typical search query terms. We therefore suggest using alternative IC corpora that are larger and better aligned with the usage of modern vocabulary. We experimentally derive two such corpora using the WWW Google search engine, and show that they provide better coverage of vocabulary, while showing comparable frequencies for Brown corpus terms. Finally, we evaluate the two proposed IC corpora in the context of a concept-based video retrieval application using the TRECVID 2005, 2006, and 2007 datasets, and we show that they increase average precision results by up to 200%. Leveraging semantic resources, such as WordNet [1] , has proven as a valuable linguistic tool in determining semantic relationships between words, sentences, and larger bodies of text [26] . WordNet's rich vocabulary and complex structure of relationships can be used to determine semantic concepts for words as well as links to near and distant related terms. With the emergence of audio-visual material and the desire to perform information retrieval on such resources, WordNet has become an important tool used in query expansion [2] . More recently, WordNet has become a medium used to bridge the gap between semantic and visual domains, as its concept classes are linked to visual counterparts [3] .. Determining the semantic similarity between words or phrases is one of the most powerful features of WordNet, and it is frequently used for query expansion purposes. Many measures for semantic similarity have been introduced, the most popular of which use some combination of WordNet's hierarchy and a measure of information content to generate a numerical score of likeness. Such measures include Wu-Palmer [4] , Resnick [5] , JiangConrath [6] , Lin [7] , Lesk [8] , and Leacock-Chodorow [9] , among others. Approaches by Resnick, Jiang-Conrath, and Lin rely heavily on the notion of information content, a numerical value which denotes a term's saliency, or importance.. A closer analysis of source of information content used with WordNet reveals that a measure of information content exists only for approx. 25% of all WordNet terms. Generally, a nonexistent value eliminates a term's semantic relatedness to any other term. In this paper, we explore the reason behind this deficiency, outline the potential negative impact, provide alternatives, and evaluate them on TRECVID search tasks [10, 11, 12] . For the work presented here, we have used WordNet version 2.1. We have presented an approach for determining information content from two web-based sources, and demonstrated its application to concept-based video retrieval. Information content is a frequently used measure, but its underlying database of term frequencies shows severe flaws due to outdated information from a too small text corpus. We show how two approaches of retrieving term frequency from web-based resources can be used to generate an improved information content database. We validate both approaches extensively, and find that the semantic similarity based on improved information content generally performs significantly better than the commonly used IC based on the Brown corpus. The WordNet lexical database models a large portion of words and phrases from the English language in a lexical hierarchy of cognitive synonyms with several types of relationships, such as hypernyms and synonyms. It is possible to resolve semantic relationships between concepts, such as the relationship between a leaf and a plant (hypernym), but also between named entities and concepts, for example, World Trade Center is "an instance of" a skyscraper and it is a type of building (hypernym).. Each of the four parts of speech captured in WordNet (noun, verb, adjective, adverb) underlies a different hierarchy of relationships. Nouns, outnumbering the other parts of speech, have the most comprehensive hierarchy, but at the same time the most straightforward linkage between terms, sharing only one common root named "entity". It is therefore possible to find a path between any two nouns, whether this path traverses the root node of "entity", or some other common sub-node. For example, the conceptually closely related terms "leaf" and "stem" share the common parent of "plant organ". Only when terms are conceptually so distant that they do not share a common subnode, such as the abstract term "thought" and the physical term "water", is the more distant root node "entity" the only link.. Unlike nouns, verbs are grouped into smaller disjoint hierarchies, each one featuring one common node. These hierarchies are very shallow compared to those of nouns, and there exist no connections between them. For instance, the verb "to walk" is grouped into a hierarchy with root node "to travel", for which no path exists to the verb "to drink" with root node "to consume".

O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O B O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O B I O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Searching for the dimension of valued preference relations The more information a preference structure gives, the more sophisticated representation techniques are necessary, so decision makers can have a global view of data and therefore a comprehensive understanding of the problem they are faced with. In this paper we propose to explore valued preference relations by means of a search for the number of underlying criteria allowing its representation in real space. A general representation theorem for arbitrary crisp binary relations is obtained, showing the difference in representation between incomparability--related to the intersection operator--and other inconsistencies--related to the union operator. A new concept of dimension is therefore proposed, taking into account inconsistencies in source of information. Such a result is then applied to each a-cut of valued preference relations. Easy decision making problems are those that allow a direct and clear answer, most probably without taking into consideration any formal abstract www.elsevier.com/locate/ijar International Journal of Approximate Reasoning 33 (2003) model, or just a simple one. The difficulty of a decision making problem is of course relative to each particular decision maker. In fact, a frequent strategy is a search for a specialist, i.e., a person we can trust who declares such a problem as easy according to his/her knowledge or experience (decision makers, or even specialists, may be wrong, and the problem is not the way they see it, but that is another problem). Difficult decision making problems usually require a formal abstract model.. In some cases, a problem cannot be understood as easy because of the amount of data or information we should keep in mind: then we need a mathematical model showing how simple the solution is. The decision maker is overwhelmed by the amount of information, but the problem requires only an organizing data procedure.. In some other cases, the formal model has a deeper role in the decision making problem, showing its internal structure. A solution will follow after some calculation. Of course we may require a second specialist in order to find a solution for the model proposed by the first specialist (although models are usually established taking into account his/her own knowledge, in such a way that the model is at least understood by the first specialist proposing such a model and therefore the existence of a solution is expected). Classical multicriteria American School and many optimization approaches can be allocated here (see [19] ): finding out the right formal model may take some time, but once such a model is accepted, we have a complete understanding of the problem and the problem itself becomes simple, even when we cannot reach a solution. We may of course be surprised by the fact that our model cannot be implemented, due perhaps to some practical restriction, and it may be even the case that we are dealing with a yet unsolved mathematical problem. Anyway, as soon as we can propose a mathematical model fitting information and our decision problem is reduced to a formal optimization problem, we can declare that our problem has been understood (until some conflicting new information reaches us).. But too often in practice we find out that our decision making problem is complex in nature: we do not expect to get a model fully explaining such a problem. We only expect to increase our knowledge about the problem, i.e., to get a better insight into the problem structure.. Simple decision making problems are those where a complete representation model has been possible. The inner structure of the problem has been fully understood, and the problem can be analyzed by means of an appropriate software playing a decision maker role. That is not the situation when the problem is complex: in real life, most people avoid those ''black boxes'' telling them what to do, partially because they know how complex in nature their problem is. Users want to be the only decision makers. It is not only a claim for an interactive procedure, which indeed will help, but the advance acknowl-edgement that there is no such complete representation of the problem. Those decision makers are then looking for a better understanding of the problem, so they can be sure they are not forgetting essential facts or possibilities (sometimes they know the solution by heart, but they still need a formal model explaining why). Such a better knowledge of the problem will hopefully open the decision makerÕs mind to new alternatives or approaches (see [16, 18] ). In this sense, classical French School and disaggregation-aggregation approaches are closer to a position for exploiting information and improve knowledge of decision makers rather than the classical American School and the multiobjective optimization approach (see [19] ).. Real complex decision making problems do need methodologies for a better understanding rather than choice proposals, i.e., aid for knowledge rather than aid for decisions. Each particular multicriteria approach can in principle be considered, in as far as each model can be showing a particular view of the problem. In this context, geometrical representation will always play a key role, as a natural way of showing elaborated information to decision makers. And it is a fact that modern multicriteria procedures give an increasing role to representation software (see, e.g., [10] ). This paper deals with such a geometrical representation. Indeed, one of the key issues in order to understand a problem is the knowledge of possible underlying criteria. Classical dimension theory [5] seems to be a natural possibility when basic information is given in terms of crisp preference relations: the number of underlying criteria may be a hint in the search of underlying criteria (to be defined later in a formal way in order to be useful). Such an approach presents well known algorithmic problems (see [23] ), partially solved in [24] .. When dealing with valued preference relations, searching for a representation that is comprehensible to the decision maker (or at least a BeltonHodgkins facilitator [2] ) is an absolute need. A first proposal can be found in [12] (see also [13] ), by exploiting information from the dimension function associated with every a-cut of a given preference relation, once some assumptions have been imposed to our valued preference relation. We now generalize such an approach, taking into account a general representation for arbitrary crisp preference relations. This paper is organized as follows: basics about classical (crisp) dimension theory are reviewed, being this dimension restricted to partial order sets (Section 2); such an approach is applied to max-min transitive valued preference relations, developing a dimension value for certain a-cuts (Section 3); a representation of arbitrary crisp preferences is then shown (Section 4), and this general result allows the definition of a dimension value for every a-cut of arbitrary valued preference relations (Section 5). Several examples are analyzed, and some particular informative indexes are proposed (Section 6).

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


On Connected Boolean Functions A Boolean function is called (co-)connected if the subgraph of the Boolean hypercube induced by its (false) true points is connected; it is called strongly connected if it is both connected and co-connected. The concept of (co-)geodetic Boolean functions is deÿned in a similar way by requiring that at least one of the shortest paths connecting two (false) true points should consist only of (false) true points. This concept is further strengthened to that of convexity where every shortest path connecting two points of the same kind should consist of points of the same kind. This paper studies the relationships between these properties and the DNF representations of the associated Boolean functions. ? A Boolean function can be viewed as a partition of the vertex set of the Boolen hypercube into a subset of "true" points and a subset of "false" points. This paper studies classes of Boolean functions for which the induced subgraph of true points and=or the induced subgraph of false points possess some special connectivity properties. For each of the classes studied in this paper, we shall concentrate our e orts on their recognition, and on their structural description.. We shall generally assume the knowledge of disjunctive normal form (DNF) representations of the functions studied, although in some of the cases we also assume the knowledge of their prime implicants.. We shall ÿrst study Boolean functions for which the subgraphs induced by their true points, or their false points, or both, are connected. We shall call these functions connected, co-connected and strongly connected, respectively. In the following part of the paper, we strengthen these conditions by requiring that at least one of the shortest paths connecting two true (false) points belongs to the subgraph induced by the true (false) points; such Boolean functions are called geodetic (co-geodetic). The strongest connectivity condition studied in this paper, convexity (co-convexity), requires that every shortest path connecting any pair of true (false) points belong to the subgraph induced by true (false) points. The deÿnitions of strongly geodetic and strongly convex functions are analogous to that of strongly connected ones.. Boolean functions having connectivity properties similar to those described above play an important role in problems appearing in various areas including in particular discrete optimization, machine learning, automated reasoning, etc.. After a brief introduction of basic concepts and notations, we study in Section 3 connected Boolean functions. This section starts with a simple graph theoretic characterization of functions whose set of true points induces a connected subgraph. This characterization uses the knowledge of all the prime implicants and is followed by a second one using an arbitrary DNF representation of the function. The section concludes by showing that recognizing the connectedness of the set of false points is CoNP-hard.. Section 4 studies various types of geodetic functions. After showing that their recognition is CoNP-hard, we show that the recognition of those functions whose true points are geodetically connected can be reduced to the solution of a satisÿability problem. The classes of monotonically non-decreasing (positive), monotonically non-increasing (negative), and more generally unate functions are all shown to have the property that both their true and false points are geodetically connected.. We introduce the class of concordant Boolean functions, generalizing the unate ones and having the property that any two of their prime implicants "con ict" in at most one variable. It is shown that concordant DNFs represent geodetic functions. Moreover, concordant functions possess the strong geodetic property that both the true and the false points are geodetically connected.. Section 5 deals with k-convex functions, i.e. functions with the property that all the shortest paths linking two true points at Hamming distance at most k consist only of true points; co-k-convex and strongly-k-convex functions are deÿned in a similar way. The ÿrst part deals with the characterization of k-convex functions and studies convexity properties of connected functions, while the second part examines problems related to their recognition.. After establishing a set of functional inequalities characterizing k-convexity, we show that the set of true points of a k-convex function in the Boolean hypercube consists of subcubes of true points at "large" Hamming distances from each other. We deÿne for every k = 2; : : : ; n, the k-convex hull of an arbitrary Boolean function as the (unique) minimal k-convex majorant of the function, and provide a polynomial algorithm for its construction.. We continue by characterizing strongly convex functions and show that any strongly convex function e ectively depends on at most one variable and that consequently the number of strongly convex functions on n variables is at most 2n + 2.. We later examine convexity properties of connected Boolean functions and show that there are only two kinds of connected functions: those which are k-convex for every k = 2; : : : ; n and those which are not k-convex for any k; analogous properties are established for co-connected and strongly connected functions.. The second part of Section 5 studies the recognition problem of various convexity properties and shows that in general these problems are computationally di cult. However, we show that for classes of functions for which a family of associated satisÿability problems can all be solved polynomially, the recognition problems become tractable; this class includes Horn, quadratic, renamable Horn and q-Horn functions. We have seen that the classes of Boolean functions examined in this paper can be represented as in Fig. 2 , where all the inclusions are proper.. We have also seen that the DNFs of a Boolean function re ect to a large extent their various connectedness properties. In spite of that, most of the recognition problems of the classes of functions studied in this paper were proven to be CoNP-hard. The only exceptions concern DNFs of connected functions, and DNFs for which some associated SAT problems are polynomially solvable. An interesting class of Boolean functions introduced in this paper is that of concordant functions, generalizing the class of unate functions. Concordant functions also include the class of acyclic renamable Horn functions and are included in the class of strongly geodetic functions.. It has been shown that the set of the true points of a k-convex function consists of entire subcubes at large Hamming distances from each other. Perhaps not suprisingly, the class of strongly k-convex functions was shown to contain very few functions; in fact, the only strongly k-convex functions are those that depend e ectively on at most one variable. Finally, it has been shown that there are only two kinds of connected functions: those which are k-convex for any k, and those which are not k-convex for any k.

O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O