Chapter I Machine Learning for Agents and Multi-Agent Systems

In order to be truly autonomous, agents need the ability to learn from and adapt to the environment and other agents. This chapter introduces key concepts of machine learning and how they apply to agent and multiagent systems. Rather than present a comprehensive survey, we discuss a number of issues that we believe are important in the design of learning agents and multi-agent systems. Specifically, we focus on the challenges involved in adapting (originally disembodied) machine learning techniques to situated agents, the relationship between learning and communication, learning to collaborate and compete, learning of roles, evolution and natural selection, and distributed learning. In the second part of the chapter, we focus on some practicalities and present two case studies. 2 Kudenko, Kazakov & Alonso Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. INTRODUCTION Intelligence implies a certain degree of autonomy, which in turn, requires the ability to make independent decisions. Truly intelligent agents have to be provided with the appropriate tools to make such decisions. In most dynamic domains, a designer cannot possibly foresee all situations that an agent might encounter, and therefore, the agent needs the ability to learn from and adapt to new environments. This is especially valid for multi-agent systems, where complexity increases with the number of agents acting in the environment. For these reasons, machine learning is an important technology to be considered by designers of intelligent agents and multi-agent systems. The goal of this chapter is not to present a comprehensive review of the research on learning agents (see Sen & Weiss, 1999, for that purpose) but rather to discuss important issues and give the reader some practical advice in designing learning agents. The organization of the chapter is as follows. In the following section, the differences between pure machine learning and that performed by (single) learning agents are discussed. We start with the introduction of basic machine learning concepts, followed by examples of machine learning techniques that have been applied to learning agents, such as Q-learning, explanation-based learning, and inductive logic programming. In the third section, we discuss several issues surrounding multi-agent learning, namely, the relationship between learning and communication; learning to collaborate and compete; the learning of roles, evolution, and natural selection; and distributed inductive learning. Following this discussion, we focus on some practicalities and present two case studies. We finish the chapter with conclusions and further work. FROM MACHINE LEARNING TO LEARNING AGENTS In this section, we discuss the nature of machine learning (ML), its integration into agents, and the parallels between machine learning systems and learning agents. We start with a basic introduction to machine learning. While most of the fundamental ML concepts introduced below are commonly associated with supervised learning (SL) (i.e., the generalization from annotated examples provided by a teacher), they are equally relevant for reinforcement learning (RL), where an agent learns through the feedback (i.e., reinforcement) from the environment in each entered state. To date, most attention in agent learning has been reserved for RL techniques such as QMachine Learning for Agents and Multi-Agent Systems 3 Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. learning (see below), due to its suitability to situated agents. Nevertheless, we see SL and RL as strongly related approaches. In the case of RL, the environment could be seen as the teacher, and the generalization process would be over states (which correspond to the examples). In fact, SL methods can be directly applied in a RL setting (see, e.g., the use of neural networks in TD learning (Tesauro, 1992). Note that we mostly exclude a third class of ML techniques, unsupervised learning (or learning by discovery), from our discussion, because there are only few research results in this area to date. Introduction to Machine Learning A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E (Mitchell, 1997). Machine learning can be seen as the task of taking a set of observations represented in a given object (or data) language and representing (the information in) that set in another language called the concept (or hypothesis) language. A side effect of this step can be the ability to deal with unseen observations. As an example, one can consider an object language that consists of pairs of coordinates (x,y). A number of observations are provided, and each is labeled as a positive or negative example of the target concept, i.e., the concept to be learned. Let the hypothesis language define a concept as an ellipse,1 such that a point (x,y) would represent a positive example of the concept if it is inside that ellipse or a negative one otherwise. In the example in Figure 1, there are infinitely many such ellipses, each of which would satisfy the usual requirements for completeness and consistency, i.e., it would cover all positive examples and none of the negative. The set of all complete and consistent hypotheses for the definition of the target concept is referred to as the version space for that concept. Selecting the Agent’s Learning Bias The hypothesis language specifies the language bias, which limits the set of all hypotheses that will be considered by the learner. In the example in Figure 1, it is obvious that there are certain arrangements of positive and negative examples on the plane that can never be separated by an ellipse in the required way. A priori, all concepts in the version space are equally good candidates for the description of the training data provided. Only when additional test data is available can one find that certain concepts in the version space classify 4 Kudenko, Kazakov & Alonso Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited. unseen members of the target concept better than others. A preference bias is a principle that allows the learner to choose between two hypotheses if they both classify the training data equally. If hypotheses from the entire version space are to be ranked according to the preference bias, the order in which they are explored is of no importance. Alternatively, a search bias can be introduced to specify that order, and the search can stop after the first acceptable hypothesis with respect to a given criterion is found. The appropriate choice of language, preference, and search biases is crucial for the learning outcome. The language bias has to be sufficiently general to include the target concept, yet restricted enough to make the search feasible. A well-chosen search bias can significantly speed the learning. At the same time, specifying the way in which the hypothesis space is searched can result in a preference bias being enforced, as in the cases when search starts with the most general, or the most concise, hypotheses. The choice of preference bias can be guided by some intuition about the type of hypothesis likely to be successful for the given type of data, or by some general principle, such as Occam’s razor, favoring the hypothesis using the minimum number of entities, i.e., the simplest or shortest one, and the Minimal Description Length (MDL) principle, which recommends “the shortest method for re-encoding the training data, where we count both the size of the hypothesis and any additional cost of encoding the data given this hypothesis” (Mitchell, 1997). Figure 1: Object and concept language. Most specific concept Most general concept