Abstract: As artificial intelligence (AI) systems are becoming increasingly autonomous and will soon be able to make decisions on their own about what to do, AI researchers have started to talk about the need to align AI with human values. The AI 'value alignment problem' faces two kinds of challenges-a technical and a normative one-which are interrelated. The technical challenge deals with the question of how to encode human values in artificial intelligence. The normative challenge is associated with two questions: "Which values or whose values should artificial intelligence align with?" My concern is that AI developers underestimate the difficulty of answering the normative question. They hope that we can easily identify the purposes we really desire and that they can focus on the design of those objectives. But how are we to decide which objectives or values to induce in AI, given that there is a plurality of values and moral principles and that our everyday life is full of moral disagreements? In my paper I will show that although it is not realistic to reach an agreement on what we, humans, really want as people value different things and seek different ends, it may be possible to agree on what we do not want to happen, considering the possibility that intelligence, equal to our own, or even exceeding it, can be created. I will argue for pluralism (and not for relativism!) which is compatible with objectivism. In spite of the fact that there is no uniquely best solution to every moral problem, it is still possible to identify which answers are wrong. And this is where we should begin the value alignment of AI.

Challenges of Aligning Artificial Intelligence with Human Values

Introduction The rapid development of artificial intelligence (AI) has enabled a wide range of beneficial applications in various contexts, from healthcare to security and governance. Fields or abilities formerly thought to be exclusively "human" are now being tackled by a machine-learning approach  (Arnold et al., 2017, p. 81) . As AI becomes more powerful, more attention is being paid to ways of maximising its societal benefit. The growing capabilities of AI and machine learning and the development of countless new applications also pose questions of risk and trust  (Sutrop, 2019) . AI could be a threat to humanity either through its malicious use by humans  (Brundage et al., 2018) , unintended consequences  (Amodei et al., 2016) , or autonomous acts by the artificial system  (Farquhar et al., 2017) . A joint report, The Malicious Use of  AI: Forecasting, Prevention, and Mitigation (2018) , issued by the leading AI research centres in the UK and the US  (Brundage et al., 2018)  has recently analysed the landscape of potential security threats from malicious uses of AI technologies and explicitly stated that if adequate defences are not developed soon, AI will threaten our digital, physical and political security. Currently, AI systems are narrowly dedicated to specific tasks such as speech, face or object recognition, spam filters or autonomous driving, and they are not capable of setting their own goals or choosing the best courses of action across domains. Researchers predict that AI will outperform humans in translating languages by 2024, writing high-school essays by 2026, driving a truck by 2027, or working as a surgeon by 2053; they believe there is a 50 per cent chance of AI outperforming humans in all tasks within 45 years and of automating all human jobs in 120 years  (Grace et al., 2018) . As robots become increasingly autonomous and able to make decisions on their own, the risk increases that humans will lose control over AI systems. Wendell  Wallach and Colin Allen (2009)  have suggested that as robots take on more and more responsibility, they must be programmed with moral decision-making abilities, for the sake of our own safety. What is AI? AI has been described in various ways. It can refer to certain humandesigned systems, or to a scientific discipline that encompasses several approaches and techniques, such as machine learning, machine reasoning, and robotics. In this paper, I will discuss AI systems according to the definition of the European Commission High Level Expert Group on Artificial Intelligence as follows: AI systems are software (and possibly also hardware) systems designed by humans that, given a complex goal, act in the physical or digital dimension by perceiving their environment through data acquisition, interpreting the collected structured or unstructured data, reasoning on the knowledge, or processing the information derived from this data and deciding the best action(s) to take to achieve the given goal. AI systems can either use symbolic rules or learn a numeric model, and they can also adapt their behaviour by analysing how the environment is affected by their previous actions.  (European Commission, 2019a)  Visionary scenarios consider the possibility that intelligence equal to our own, or even exceeding it, will already be created within the current century. An AI equipped with human-level or even greater ability, which is able to find a solution when presented with an unfamiliar task, is referred to as Artificial General Intelligence (AGI). Indeed, freed from biological constraints such as limited memory and slow biochemical processing speeds, machines may eventually become more intelligent than we are-with profound implications for us all. It has been hypothesised that if humans can create AGI at a human level of intelligence, such an innovation could give rise to higher and higher intelligence; in combination with recursive self-improvement of the AGI, this would eventually lead to Superintelligence (SAI)  (Kurzweil, 2005; Tegmark, 2017) . SAI is "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest"  (Bostrom, 2014, p. 26) . According to a high-level survey of expert opinion, AI specialists estimate that systems will move to superintelligence by the end of this century, and that "the chance is about one in three that this development turns out to be 'bad' or 'extremely bad' for humanity"  (Müller & Bostrom, 2016) . Even if we are still far from humanlevel AI and AI advancements may be overhyped, several research programs have been launched on risk assessment of AI systems, attempting to mitigate threats of rapid and uncontrollable AI development. The AI developers themselves worry about how to ensure that an AI does not escape its confines and cause damaging effects  (Yudkowsky, 2004) . Computer scientists warn, "[a]s autonomous systems become more prevalent in society, it becomes increasingly important that they robustly behave as intended"  (Russell et al., 2016) . Ethicists stress, "[w]hen independent of the humans that created them, their true 'intelligence' is tested in terms of their status as 'moral beings'"  (Iphofen & Kritikos, 2019) . Since the potential threats are great, policymakers have emphasised the need to develop ethically-aligned AI which will respect our ethical principles and values. A good overview of various guidelines for ethical, rights-respecting and beneficial AI has been provided in the recent report 'Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-based Approaches to Principles for The research program of designing AI that conforms to human values is called 'value alignment'. AI researchers explain this as follows: "As intelligent systems gain autonomy and capability, it becomes vital to ensure that their objectives match those of their human users; this is known as the value alignment problem"  (Fisac et al., 2018, p. 1) . 'Value alignment' is defined as a property of an intelligent agent that allows it only to pursue goals and activities which are beneficial to humans  (Soares & Fallenstein, 2014; Soares, 2015; Russell et al., 2016; Arnold et al., 2017; Russell, 2019) . Identifying the purposes we really desire remains an open question. The opposite of 'value alignment', 'value misalignment' can be explored from two perspectives, either by focusing on the behavior of individual tech artefacts (individual level) or on the functioning of sociotechnical systems (system-level)  (Osoba et al., 2020, p. 332) . The AI 'value alignment' problem faces two interrelated kinds of challengestechnical and normative  (Gabriel, 2020) . The technical challenge deals with the question of "how to encode human values in artificial intelligence." The normative challenge is associated with two questions: "which values or whose values should artificial intelligence align with?" My concern is that AI developers and computer scientists underestimate the difficulty of answering the normative question. They hope that if we find out "what we want, what we really, really want"  (Russell, 2015) , we can create a beneficial AI which is "humble, altruistic, and committed to pursue our objectives, not theirs". But how are we to decide which objectives or values to induce in AI, given that there is a plurality of values and moral principles and that our everyday life is full of moral disagreements? In the first part of my paper I will demonstrate how technical and normative challenges of value alignment are interrelated. I will provide a brief overview of the ways in which AI researchers have tried to solve the 'value alignment problem'. I will explain why it is problematic that in value alignment programs the word 'value' is used interchangeably with goals or preferences. In the second part of my paper I will focus on normative challenges of value alignment. Several research papers, AI guidelines and manuals discuss the need to take into account cultural differences, while also respecting universal principles. I will suggest how this apparent inconsistency can be removed. My thesis is that although it is not realistic to reach an agreement on what we humans really want, given that people value different things and seek different ends, it may be possible to agree on what we do not want to happen, considering the possibility that intelligence can be created that is equal to our own, or even exceeding it. I agree with Isaiah Berlin's view as expressed in 'The Pursuit of the Ideal' that even if our ends and moral principles are plural, they are not infinitely many, since they have to remain "within the human horizon"  (Berlin, 2013 (Berlin, [1947 ). I will argue for pluralism (and not for relativism!) which is compatible with objectivism. Despite the fact that there is no uniquely best solution to every moral problem, it is still possible to identify which answers are wrong. And this is where we should begin the value alignment of AI.
 A technical challenge: how to align AI with values? The answer to the question how to align AI with values depends, on the one hand, on the technical possibilities and, on the other hand, on what we want AI to align with. Theories of value alignment vary on what they mean by 'value'. In the context of value alignment, Iason  Gabriel (2020)  notes that the notion of 'value' can serve as a place-holder for many things: AI could be designed to align with expressed or intended instructions, revealed preferences, informed preferences or desires, interest or well-being, or moral values as defined by the individual or society. In view of these different approaches, value alignment turns out to be an umbrella for various attempts to ensure that general artificial intelligence will remain beneficial to humans. Through what means do AI researchers and designers hope to reach value alignment? There seem to be two basic-albeit opposite-approaches, either feeding the artificial agent the right principles or applying (cooperative inverse) reinforcement learning. Both approaches-feeding the artificial intelligent with principles and reinforcement learning-have their proponents and critics. With progress in the area of machine learning, the idea of training a system on data (either supervised or unsupervised) has drawn increasing attention. Thus there seems to be a turn away from traditional machine ethics with its topdown articulation of rules, norms or models of behaviour and toward learning values either through imitation or by identifying the preferences of humans. Nevertheless, some critics doubt that beneficial AI can be achieved in this way, arguing that AI can be truly beneficial only if it can become a moral agent which acts on ethical principles or has moral virtues. Iason  Gabriel (2020)  has recently argued for a principle-based approach. His view is that instead of looking for "true" moral principles for AI, the real task is "to identify fair principles for alignment that receive reflective endorsement despite widespread variation in people's moral beliefs"  (Gabriel, 2020) . The idea behind reinforcement learning is that desirability of any state sequence can be expressed as a sum of immediate rewards associated with each state in the sequence. The objectives that the reinforcement learning agent is supposed to follow can be defined by specifying reward function. In the case of inverse reinforcement learning (IRL), an AI system infers the underlying utility function of an agent by observing its behavior.  Hadfield-Menell et al. (2016)  argue that the value alignment problem can only be solved through cooperative inverse reinforcement learning (CIRL) which they describe as "a cooperative, partialinformation game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is." Mark O.  Riedl and Brent Harrison (2016)  have argued that given potentially infinite undesirable outcomes in an open world, if values cannot be easily enumerated by human programmers, they can be learned by reinforcement. They hypothesise that an artificial intelligence that can read and understand stories can learn the values tacitly held by the culture from which the stories originate.  Riedl and Harrison (2016)  describe their preliminary work on using stories to generate a value-aligned reward signal for reinforcement learning agents that preempts psychotic-appearing behavior. They argue, Value alignment in a reinforcement learning agent theoretically can be achieved by providing the agent with a reward signal that encourages it to solve a given problem and discourages it from performing any actions that would be considered harmful to humans. A value-aligned reward signal will reward the agent for doing what a human would do in the same situations when following social and cultural norms (for some set of values in a given society and culture) and penalise the agent if it performs actions otherwise. A reinforcement learning agent will learn that it cannot maximise reward over time unless it conforms to the norms of the culture that produced the reward signal.  (Riedl & Harrison, 2016)  How can a value-aligned reward signal be learned?  Riedl and Harrison's (2016)  answer is that, "[a] value-aligned reward signal is produced from the crowdsourced stories in a two-stage process". When humans tell stories to other humans, they skip over many details and events. By crowdsourcing the narratives, one can get better coverage of all steps involved in a situation and extract the most reliable pattern of events. Riedl and Harrison describe the reinforcement learning process as follows: in the first step, the plot graph learning process "aligns" the crowdsourced example narratives to extract the most reliable patterns of events; in the second step, the plot graph is translated into a "trajectory tree" which is used to produce a reward signal. If the reinforcement learning agent performs an action that is a successor of the current node in the trajectory tree, it receives a reward; if it fails, it receives a small punishment. The aim is to make the reinforcement learning agent solve some problem in the most human-like fashion. The problem is that in many stories humans violate social and cultural norms. The authors' hope is that the most typical human behaviour that emerges from crowdsourced stories is compliant to the norms. The proposal to achieve value alignment through reinforcement learning has been criticised by Thomas Arnold, Daniel Kasenberg and Matthias Scheutz (2017) as ethically inadequate and insufficient to guide artificial agents in decision making. They point out a number of problematic areas associated with reinforcement learning, including data bias, generalisation issues, and the adequacy of reward functions to represent temporally complex norms  (Arnold et al., 2017, p. 81) .  Arnold et al. (2017)  claim that even if there is agreement that top-down approaches of giving AI systems ethics would be intractable, IRL on its own cannot solve the problem and train such an agent to be moral. They point out that ethical behavior depends not only on the acts themselves, but on intentions, reasons, norms, and counterfactuals. The question IRL faces is what kind of model could truly learn ethical practices. Alternatively, value alignment of AI can take the form of imitation learning. In their article entitled 'The virtuous machine-old ethics for new technology' Nicolas  Berberich and Klaus Diepold (2018)  draw inspiration from Aristotle to demonstrate that virtue ethics can provide a solution to the value alignment problem. Instead of focusing on the formulation of duties and maxims (as deontologists do) or on the identification of desirable and maximisable consequences of actions (i.e., happiness), Berberich and Diepold argue that one should concentrate on moral actions performed by virtuous moral agents. "With respect to machines this changes the primary question from 'By what kind of algorithm can a machine choose the right action?' to 'How can we build a machine that, owing to its constitution, acts appropriately in arbitrary situations?'"  (Berberich & Diepold, 2018, p. 4 ). The article concludes by stating, 1. Virtue ethics fits nicely with modern artificial intelligence research and is a promising moral theory. 2. Taking the virtue ethics route to building moral machines allows for a much broader approach than simple decision-theoretic judgement of possible actions. Instead, it takes other cognitive functions into account like attention, emotions, learning and actions.  (Berberich & Diepold, 2018, p. 23)  Berberich and Diepold believe that since humans have for centuries learned virtues from imitating the behaviour of virtuous moral agents, virtue theory should be the guiding moral theory for building truly moral machines. They point out that since virtues are an integral part of one's character, the AI would not have the desire of changing its virtue of temperance  (Berberich & Diepold, 2018, pp. 23-24) . The difficulty in this argument is that virtue ethics has a hard time giving reasons for actions, and this means one cannot claim responsibility. The most influential approach to the problem of AI value alignment has been provided by Stuart Russell, whose ideas are summarised in his recent book, Human Compatible: AI and the Problem of Control (2019).  Russell's (2019)  central idea is that in order to maintain control, we have to design beneficial AI which is "humble, altruistic, and committed to pursue our objectives, not theirs". In order to guide AI researchers and developers in thinking about how to create beneficial AI systems, Stuart Russell formulates three principles: 1. The machine's only objective is to maximise the realization of human preferences. 2. The machine is initially uncertain about what those preferences are. 3. The ultimate source of information about human preferences is human behavior.  (Russell, 2019, pp. 172-173)  Russell believes that there is plenty of data about human actions (what has been written, filmed or observed) and about attitudes to these actions. In other words, rather than having a detailed ethical taxonomy programmed into them, AI systems should infer human values by observing and emulating our behavior. It is noteworthy that in his earlier writings and interviews, Russell has spoken about 'human values' which have now been replaced by the expression 'human preferences'. In an interview from 2015, Russell stated: To the extent that human values are shared, machines can and should share what they learn about human values. […] by assigning very broad priors over what human values might be, and by making the AI system risk-averse, it ought to be possible to induce exactly the behavior one would want: before taking any serious action affecting the world, the machines engage in an extended conversation with us and an extended exploration of our literature and history to find out what we want, what we really, really want.  (Russell, 2015)  In his 2019 book, Russell replaces the expression 'human values' with 'human preferences'. He also provides explanations as to why he has dropped the word 'value'. Russell points out that since the word 'value' is often understood as 'moral value', use of this term can be confusing. He does not want to give an impression of attempting to solve moral dilemmas like stops on a trolley route. In his view, such an approach is not worth considering, as in true dilemmas there are good arguments for both sides, and artificial agents cannot cause more harm than humans even if they take a wrong decision. Russell's concern is how to avoid the worst catastrophes and to ensure that we never lose control of machines more intelligent than we are. In addition, Russell insists that one should not equate value alignment with the wish to instill in machines a single ideal value system which may raise the question whose values should be encoded or who has been given the right to align AI with such values. He says that he uses the word 'value' as a technical term, "synonymous with utility, which measures the degree of desirability of anything from pizza to paradise"  (Russell, 2019, p. 178) . He also explains what he means by 'preferences': these cover everything one might care about. Furthermore, in order to be beneficial, the purely altruistic machine should aim to maximise the realisation of human preferences. Instead of "putting in values" or preferences, the machines "should learn to predict better, for each person, which life that person would prefer, all the while being aware that the predictions are highly uncertain and incomplete"  (Russell, 2019, p. 178 ). Russell believes that, in principle, an AI could learn billions of different predictive preference models. With respect to the issue of how the machine should trade off the preferences of multiple humans,  Russell (2019, p. 174 ) refers to the utilitarian principle of "the greatest happiness for the greatest numbers". The crucial question is how the machine might learn about human preferences. His answer is, again, very simple: by observing human choices which reveal information about human preferences. In Russell's opinion, the real complications arise because humans are not perfectly rational and the machine must take such an imperfection into account  (Russell, 2019, p. 177) . It turns out that, in essence, what Russell means by 'value' is in reality a 'preference'. However, the same problem crops up with the concept of 'preference' as it did with the concept of 'value'. Both have multiple meanings, and their use in different disciplines is highly variable. In its most general meaning, a value is that which is worthy, that which is worth having, getting or doing, or that which possesses some property or properties that make it so. Values are action-guiding; they are taken into account when making decisions and when planning activities. In the most general terms, preferences are evaluations; they concern matters of value. Preferences are subjective in that the evaluation is typically attributed to an agent. Russell uses the term as subjective comparative evaluation, of the form 'Agent A prefers X to Y'. What one prefers is not necessarily something objectively valuable, but rather something subjectively valuable from one's personal point of view. But this is where the problem begins. We can imagine the situation where the designer of a self-driving car asks potential customers whether they would prefer a car which will protect the persons riding in it or those on the street. Or, even worse, whether they would like a car which violates traffic rules, ignores traffic lights and brings the owner quickly and safely to the destination. Should such preferences be fulfilled by artificial intelligence? By leaving aside moral values and concentrating only on human preferences, it is difficult to understand how situations in which AI might fulfil the preferences of immoral agents can be avoided. As there are more non-democratic countries than democratic ones in the world, we should be careful not to allow AI to be programmed to do something devastating. Somewhere there will always be efforts underway to create an AI that will be destructive, for example, for military purposes. Some experts have questioned the use of robots in military combat, especially if such machines were to be given some degree of autonomous functions, e.g., being able to independently choose targets to attack with weapons. What is the solution? AI should be programmed not to fulfil immoral preferences and refuse to do immoral actions but they should be able to distinguish between moral and immoral actions. Thus, besides values we should also provide them with the capacity of moral deliberation to decide when a certain moral principle applies, and to be able to rank values in different contexts. It seems dangerous to let AI become an autonomous decision-maker without its becoming a moral agent. To be a moral agent entails far more criteria than adopting ethical values. Amitai  Etzioni and Oren Etzioni (2018)  note that to be a moral agent requires a certain set of attributes: (a) Self-consciousness. If the agent is not aware of itself in any given situation, and of the alternative courses that might be followed, then no moral decisions can be rendered. (b) The agent must be aware that she can affect the situation. (c) The agent must be able to understand the moral principles to be employed in arriving at a particular moral choice. (d) The agent must have a motive to act morally. This involves having passions, as otherwise moral preferences are merely intellectual preferences with nothing to fuel the moral action. Some scholars also add that a will or intention is required…  (Etzioni & Etzioni, 2018, pp. 239-240) . This argumentation indicates that if we want to avoid the scenario that AI can satisfy both moral and immoral preferences, we need to integrate ethics into AI, not only align it with values. The research programme of AI value alignment should ideally be interdisciplinary, besides computer scientists also involving philosophers, lawyers, economists, and cognitive scientists, psychologists and pedagogical scientists. Its success will depend on how much the AI designers will be able to integrate the knowledge provided by these disciplines in the design of AI. So far, discussions on value alignment have not actively included ethicists. As a result, there is a danger that AI ethics will undergo simplification, reduction or dogmatism. In several working groups which aim to secure beneficial AI, ethics has been boiled down to law or reduced to some specific principle (fairness, transparency, equality or accountability) without providing any reasons why one or another principle should be prioritised. The analysis of the policy documents on ethical AI  (Fjeld et al., 2020)  showed that the most common governance regime suggested for dealing with AI is international human rights law-64 per cent of all documents contained a reference to human rights. Annette  Zimmermann and Bendert Zevenbergen (2019)  have recently issued warnings concerning seven traps into which AI ethics can fall: reductionism, oversimplification, the relativism trap, value-alignment trap, dichotomy; the myopia trap; and the trap of the rule of law. For the purposes of this paper, it is especially interesting how the authors present the relativism trap and the valuealignment trap as two unacceptable options. The relativism trap is associated with the tendency to conclude that since pervasive moral disagreements exist, ethics is relative. Should we argue that relativism is wrong, we risk falling into the value-alignment trap according to which there must be one morally right answer. If we cannot reach an agreement, we will have failed. I am quite inclined to agree with the authors' view that AI ethics should welcome value pluralism without collapsing into extreme value relativism. We should accept that it may not always be possible to determine "the one right answer", thereby avoiding moral disagreement. However, it is often possible to clarify that at least some paths of action are clearly wrong and some paths of action are comparatively better. The normative challenge: which values should AI align with? In order to answer the normative question 'Which values should AI align with?' one must first answer the question of whether values are objective or subjective; universal or culturally relative. Are there absolute values or prima facie values? Several regulatory bodies that write guidelines about ethical AI stress that one has to be sensitive to cultural differences while at the same time respecting universal principles. Thus the Ethics Guidelines for Trustworthy AI (European Commission, 2019b) assert that "AI systems should respect the plurality of values and choices of individuals", thereafter claiming that "certain fundamental rights and principles, such as human dignity, are absolute and cannot be subject to a balancing exercise" (European Commission, 2019b, p. 13). In order to make it possible for European citizens to reap the benefits of AI, it needs to be "aligned with our foundational values of respect for human rights, democracy, and rule of law" (European Commission, 2019b, p. 4). IBM's manual entitled Everyday Ethics for Artificial Intelligence states: "Care is required to ensure sensitivity to a wide range of cultural norms and values. As daunting as it may seem to take value systems into account, the common core of universal principles is that they are a cooperative phenomenon." (IBM, 2019) Another important document, Ethically Aligned Design. A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems, published under the auspices of the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems poses the rhetorical question: "If machines engage in human communities as autonomous agents, then those agents will be expected to follow the community's social and moral norms. A necessary step in enabling machines to do so is to identify these norms. But whose norms?" (IEEE, 2019) Again, it seems that in spite of stressing the variety of social and moral norms, this document emphasises that values-based design of AI should respect universal ethical principles and internationally recognised human rights. Indeed, the document states that […] whether our ethical practices are Western (e.g., Aristotelian, Kantian), Eastern (e.g., Shinto, School of Mo, Confucian), African (e.g., Ubuntu), or from another tradition, honoring holistic definitions of societal prosperity is essential versus pursuing one-dimensional goals of increased productivity or gross domestic product (GDP). Autonomous and intelligent systems should prioritize and have as their goal the explicit honoring of our inalienable fundamental rights and dignity as well as the increase of human flourishing and environmental sustainability.  (IEEE, 2019, p. 1)  It seems to me that all these documents point in the same direction-namely, that we should accept plurality of values while at the same time prioritising certain specific values, and doing both of these things without necessarily providing explanations of how pluralism of values and objectivity of ethics are compatible. In the following I will try to explain how pluralism can accommodate both persistent moral disagreements and the objectivity of ethics. Pluralism differs from both relativism and subjectivism. Unlike the relativist, who believes that rightness depends on the culture or society, and unlike the subjectivist, who believes that rightness depends on the individual, the pluralist holds that in some cases the question of what is right lacks a determinate answer. Rightness is indeterminate, as there is irreducibility of values. As shown by Susan Wolf in the article 'Two levels of pluralism  ' (1992) , this irreducibility can happen on two levels of decision: on the level of values and on the level of moral systems. Wolf says that on the first level we constrain moral judgements within a moral system, i.e., we have to decide which value, for example, equality or liberty, is more important and we have to admit that there is no right answer to this question. On the second level, we constrain moral assessments of the systems themselves. Here we have to decide which moral system, with its different understanding of moral rightness, we should prefer (for example, Kantianism or utilitarianism or virtue ethics).  Wolf (1992)  shows that there is a possibility that after a careful assessment it may turn out that these value systems are all incommensurably good, and living according to any of these moral systems will lead to a morally good life. Also Isaiah Berlin supports pluralism which is compatible with objectivism. Pluralism means for him that "there are many different ends that men may seek and still be rational, fully men, capable of understanding each other"  (Berlin 2013 (Berlin [1947 , p. 11). In fact, we do have shared values which form bridges between people and make intercommunication between cultures in time and space even possible. "But our values are ours, and theirs are theirs"  (Berlin 2013 (Berlin [1947 , p. 11). We might criticise the values of other cultures, but we have to understand where they come from, what it means to live the way they do. Here Berlin emphasises people's empathy and capacity for imagination, their need to put themselves in the shoes of a representative of another era or culture. Pluralism in Berlin's view does not exclude a world of objective values or ends that men pursue for their own sake. Ends and moral principles are plural, but they are not infinitely many, since they have to remain "within the human horizon"  (Berlin 2013 (Berlin [1947 , p. 12). We should be able to distinguish between objectively good and bad life forms and condemn slavery, sexism, racism, or Nazism. Given that there are people who argue from an egoistic position of power, with the intention of preserving their preferred state or that some lack imagination and empathy for what it would mean to be in another's situation (in cases such as slavery, Nazism, female circumcision, racial or gender hatred, disparagement of minorities, abuse of animals), we need to have the opportunity to say that such things are unacceptable. This remains so even if we favor value pluralism. We can get a better idea of how pluralism can be compatible with objectivity by reading John Kekes's book The Morality of  Pluralism (1993) . Kekes makes a distinction between primary and secondary values. Primary values are the same for everyone, for they constitute the minimum requirements of good lives and are derivable from the facts of human nature. For example, we share the same physiological and psychological needs and capacities. Thus, the same things, goods pertaining to the self, intimacy, and social order are needed by everybody to live a good life. At the same time, the identity of secondary values depends on variable social and personal circumstances. For example, whereas killing is universally considered to be evil, what sort of killing counts as murder may be dependent on context. Kekes explains that secondary values are contingent on primary ones; either they are particular forms of expression of these values (for instance, the primary value of intimacy will take the form of certain sexual practices) or they are genuinely new goods, the realisation of which depends on the presence of primary values (for instance, in order to derive pleasure from creative participation in art or science, one's elementary needs for food and shelter need to be satisfied)  (Kekes, 1993, pp. 42-43) . Primary and secondary values can be either moral or non-moral. While moral values (both primary and secondary) affect others, nonmoral values (both primary and secondary) affect the agent or occur naturally. Primary values (both moral and non-moral) are universal requirements of all good lives, while secondary values count as benefits or harms because a tradition or a conception of a good life makes them so  (Kekes, 1993, p. 45) . Kekes explains that pluralists differ from relativists in holding primary values to be objectively justifiable by referring to our common nature. He admits that even monists must not reject plurality of values, provided there is some overriding value which allows authoritative ranking of the plural values. Kekes points out that there is a great variety of monist conceptions of what the overriding value or some principle of ranking values may be: the utilitarian ideal of the greatest happiness for the greatest number of people, the welfare economists' goal of preference satisfaction, the Kantian principle of the categorical imperative, the contractarian list of fundamental human rights, or something else  (Kekes, 1993, p. 47) . Thus, the main difference between monists and pluralists is that for the former there is one overriding value or principle, while for the latter values are manifold and contextual. Kekes explains that we usually aim to construct a conception of the good life out of the primary and secondary values that we ourselves favor and that our tradition supplies. Although he argues that for pluralists there is no authoritative way of settling the value conflict, there still seems to be at least one overriding principle for comparing and ranking different values. To my mind, by arguing that primary values originate in our common human nature, Kekes is also suggesting an authoritative overriding principle which helps to compare and rank values. For instance, he proposes that in value conflicts one should try to reduce secondary values to primary values (which have their source in human nature), by making it explicit that in spite of all differences, the fact that situatedness in various historically conditioned traditions and individual life-situations has led to differentiation in our conceptions of a good life, we continue to have similar human needs and capacities. Although Kekes himself has not recognised that his own appeal to common human nature is monistic in a similar way as the appeal to human rights framework, categorical imperative or general happiness principle, it is still possible that he ultimately speaks for pluralism and not monism. I think that as long as one accepts that there can be a plurality of ultimate values, all of which can be equally justified, one is a pluralist. Thus, indeed, pluralism really is compatible with objectivism.
 Conclusions I have tried to show that it is good that AI researchers have identified the importance of ensuring that AI will remain beneficial and that therefore AI should align with human values. However, since the term 'value' only functions as a placeholder and in reality one could limit oneself to making AI fulfil human preferences, the problem may arise that AI will not necessarily act morally, but potentially also satisfy immoral preferences. Thus, in order to secure ethical AI one should compel it to follow moral principles or obtain moral values. In the course of the argument the question emerged whether there are universal values or whether AI should reckon with cultural differences. I hope I have shown that value pluralism can accommodate both moral disagreements caused by diversity of values and the objectivity of ethics, accepting that there are no uniquely correct answers to moral questions. Moral ambiguity does not necessarily imply relativism. Accepting plurality does not mean that anything goes. We can identify wrong answers as long as we share our expectations of morality. Indeed, the aim of morality is to regulate our life in society, enabling us to live together peacefully and allowing human beings to satisfy their various needs: physiological, psychological, social, cultural and intellectual.  Thus, value alignment should begin with seeking agreement on negative valueswhat we do not want to lose and what we do not want to happen if it becomes possible to create artificial intelligence with intellectual capacities equal to or beyond our own. Our efforts to define what is immoral should keep in mind that morality's function is to satisfy the basic human needs which we all share.Value plurality and objectivity can be joined if we accept that there are many moral principles by which one can justify and rank moral values. Different moral theories can prefer different ultimate principles. One can live a moral life according to all of these. Nevertheless, there are fewer choices about what is morally wrong, and thus we should seek agreement on what is clearly immoral.


Abstract: The area of learning in multi-agent systems is today one of the most fertile grounds for interaction between game theory and artificial intelligence. We focus on the foundational questions in this interdisciplinary area, and identify several distinct agendas that ought to, we argue, be separated. The goal of this article is to start a discussion in the research community that will result in firmer foundations for the area. 1

If multi-agent learning is the answer, what is the question?

Introduction The topic of learning in multi-agent systems, or multi-agent learning (MAL henceforth), has a long history in game theory, almost as long as the history of game theory itself.  2  As early as 1951, fictitious play  [10]  was proposed as a learning algorithm for computing equilibria in games and there have been proposals for how to evaluate the success of learning rules going back to  [23]  and  [5] . Since that time hundreds, if not thousands, of articles have been published on the topic, and at least two books (  [20]  and  [54] ). In Artificial Intelligence (AI) the history of single-agent learning is as rich if not richer, with thousands of articles, many books, and some very compelling applications in a variety of fields (for some examples see  [29, 40] , or  [50] ). While it is only in recent years that AI has branched into the multi-agent aspects of learning, it has done so with something of a vengeance. If in 2003 one could describe the AI literature on MAL by enumerating the relevant articles, today this is no longer possible. The leading conferences routinely feature articles on MAL, as do the journals.  3  While the AI literature maintains a certain flavor that distinguishes it from the game theoretic literature, the commonalities are greater than the differences. Indeed, alongside the area of mechanism design, and perhaps the computational questions surrounding solution concepts such as the Nash equilibrium, MAL is today arguably one of the most fertile interaction grounds between computer science and game theory. The MAL research in both fields has produced some inspiring results. We will not repeat them here, since we cannot be comprehensive in this article, but nothing we say subsequently should be interpreted as belittling the achievements in the area. Yet alongside these successes there are some indications that it could be useful to take a step back and ask a few basic questions about the area of MAL. One surface indication is the presence of quite a number of frustrating dead ends. For example, the AI literature attempting to extend Bellman-style single-agent reinforcement learning techniques (in particular, Q-learning  [53] ) to the multi-agent setting, has fared well in zero-sum repeated games (e.g.,  [36]  and  [38] ) as well as common-payoff (or 'team') repeated games (e.g.,  [14, 31, 52] ), but less well in general-sum stochastic games (e.g.,  [21, 26, 37] ) (for the reader unfamiliar with this line of work, we cover it briefly in Section 4). Indeed, upon close examination, it becomes clear that the very foundations of MAL could benefit from explicit discussion. What exact question or questions is MAL addressing? What are the yardsticks by which to measure answers to these questions? The present article focuses on these foundational questions. To start with the punch line, following an extensive look at the literature we have reached two conclusions: • There are several different agendas being pursued in the MAL literature. They are often left implicit and conflated; the result is that it is hard to evaluate and compare results. • We ourselves can identify and make sense of five distinct research agendas. Not all work in the field falls into one of the five agendas we identify. This is not necessarily a critique of work that doesn't; it simply means that one must identify yet other well-motivated and well defined problems addressed by that work. We expect that as a result of our throwing down the gauntlet additional such problems will be defined, but also that some past work will be re-evaluated and reconstructed. Certainly we hope that future work will always be conducted and evaluated against well-defined criteria, guided by this article and the discussion engendered by it among our colleagues in AI and game theory. In general we view this article not as a final statement but as the start of a discussion. In order to get to the punch line outlined above, we proceed as follows. In the next section we define the formal setting on which we focus. In Section 3 we illustrate why the question of learning in multi-agent settings is inherently more complex than in the single-agent setting, and why it places a stress on basic game theoretic notions. In Section 4 we provide some concrete examples of MAL approaches from both game theory and AI. This is anything but a comprehensive coverage of the area, and the selection is not a value judgment. Our intention is to anchor the discussion in something concrete for the benefit of the reader who is not familiar with the area, and-within the formal confines we discuss in Section 2-the examples span the space of MAL reasonably well. In Section 5 we identify five different agendas that we see (usually) implicit in the literature, and which we argue should be made explicit and teased apart. We end in Section 6 with a summary of the main points made in this article. A final remark is in order. The reader may find some of the material in the next three sections basic or obvious; different readers will probably find different parts so. We don't mean to insult anyone's intelligence, but we err on the side of explicitness for two reasons. First, this article is addressed to at least two different communities with somewhat different backgrounds. Second, our goal is to contribute to the clarification of foundational issues; we don't want to be guilty of vagueness ourselves.
 The formal setting We will couch our discussion in the formal setting of stochastic games (aka Markov games). Most of the MAL literature adopts this setting, and indeed most of it focuses on the even more narrow class of repeated games. Furthermore, stochastic games also generalize Markov Decision Problems (MDPs), the setting from which much of the relevant learning literature in AI originates. These are defined as follows. A stochastic game can be represented as a tuple: (N, S, A, R, T ). N is a set of agents indexed 1, . . . , n. S is a set of n-agent stage games. A = A 1 , . . . , A n , with A i the set of actions (or pure strategies) of agent i (note that we assume the agent has the same strategy space in all games; this is a notational convenience, but not a substantive restriction). R = R 1 , . . . , R n , with R i : S × A → R giving the immediate reward function of agent i for stage game S. T : S × A → Π(S) is a stochastic transition function, specifying the probability of the next stage game to be played based on the game just played and the actions taken in it. We also need to define a way for each agent to aggregate the set of immediate rewards received in each state. For finitely repeated games we can simply use the sum or average, while for infinite games the most common approaches are to use either the limit average or the sum of discounted awards ∞ t=1 δ t r t , where r t is the reward received at time t. A repeated game is a stochastic game with only one stage game, while an MDP is a stochastic game with only one agent. While most of the MAL literature lives happily in this setting, we would be remiss not to acknowledge the literature that does not. Certainly one could discuss learning in the context of extensive-form games of incomplete and/or imperfect information (cf.  [28] ). We don't dwell on those since it would distract from the main discussion, and since the lessons we draw from our setting will apply there as well. Although we will not specifically include them, we also intend our comments to apply at a general level to large population games and evolutionary models, and particularly replicator dynamics (RD)  [47]  and evolutionary stable strategies (ESS)  [49] . These are defined as follows. The replicator dynamic model assumes a population of homogeneous agents each of which continuously plays a two-player game against every other agent. Formally the setting can be expressed as a tuple (A, P 0 , R). A is the set of possible pure strategies/actions for the agents indexed 1, . . . , m. P 0 is the initial distribution of agents across possible strategies, m i=1 P 0 (i) = 1. R : A × A → R is the immediate reward function for each agent with R(a, a ) giving the reward for an agent playing strategy a against another agent playing strategy a . The population then changes proportions according to how the reward for each strategy compares to the average reward: dt (P t (a)) = P t (a)[u t (a) − u * t ], where u t (a) = a P t (a )R(a, a ) and u * t = a P t (a)u t (a). A strategy a is then defined to be an evolutionary stable strategy if and only if for some > 0 and for all other strategies a , R(a, (1 − )a + a ) > R(a , (1 − )a + a ). As the names suggest, one way to interpret these settings is as building on population genetics, that is, as representing a large population undergoing frequent pairwise interactions. An alternative interpretation however is as a repeated game between two agents, with the distribution of strategies in the population representing the agent's mixed strategy (in the homogeneous definition above the two agents have the same mixed strategy, but there exist more general definitions with more than two agents and with non-identical strategies). The second interpretation reduces the setting to the one we discuss. The first bears more discussion, and we do it briefly in Section 4. And so we stay with the framework of stochastic games. What is there to learn in these games? Here we need to be explicit about some aspects of stochastic games that were glossed over so far. Do the agents know the stochastic game, including the stage games and the transition probabilities? If not, do they at least know the specific game being played at each stage, or only the actions available to them? What do they see after each stage game has been played-only their own rewards, or also the actions played by the other agent(s)? Do they perhaps magically see the other agent(s)' mixed strategy in the stage game? And so on. In general, games may be known or not, play may be observable or not, and so on. We will focus on known, fully observable games, where the other agent's strategy (or agents' strategies) is not known a priori (though in some case there is a prior distribution over it). In our restricted setting there two possible things to learn. First, the agent can learn the opponent's (or opponents') strategy (or strategies), so that the agent can then devise a best (or at least a good) response. Alternatively, the agent can learn a strategy of his own that does well against the opponents, without explicitly learning the opponent's strategy. The first is sometimes called model-based learning, and the second modelfree learning. In broader settings there is more to learn. In particular, with unknown games, one can learn the game itself. Some will argue the restricted setting is not a true learning setting, but (a) much of the current work on MAL, particularly in game theory, takes place in this setting, and (b) the foundational issues we wish to tackle surface already here. In particular, our comments are intended to also apply to the work in the AI literature on games with unknown payoffs, work which builds on the success of learning in unknown MDPs. We will have more to say about the nature of 'learning' in the setting of stochastic games in the following sections.
 On some special characteristics of multi-agent learning Before launching into specifics, we wish to highlight the special nature of MAL. There are two messages we would like to get across, one aimed at AI researchers specifically and one more broadly. Both lessons can be gleaned from simple and well-known examples. Consider the game described in Fig.  1 . In this game the row player has a strictly dominant strategy, Down, and so seemingly there is not much more to say about this game. But now imagine a repeated version of this game. If the row player indeed repeatedly plays Down, assuming the column player is paying any attention, he (the column player) will start responding with Left, and the two will end up with a repeated (Down, Left) play. If, however, the row player starts repeatedly playing Up, and again assuming the column player is awake, he may instead start responding by playing Right, and the two players will end up with a repeated (Up, Right) play. The lesson from this is simple yet profound: In a multi-agent setting one cannot separate learning from teaching. In this example, by playing his dominated strategy, the row player taught the column player to play in a way that benefits both. Indeed, for this reason it might be more appropriate to speak more neutrally about multi-agent adaptation rather than learning. We will not fight this linguistic battle, but the point remains important, especially for computer scientists who are less accustomed to thinking about interactive considerations than game theorists. In particular, it follows there is no a priori reason to expect that machine learning techniques that have proved successful in AI for single-agent settings will also prove relevant in the multi-agent setting. The second lesson we draw from the well-known game of Rochambeau, or Rock-Paper-Scissors, given in Fig.  2 . As is well known, this zero-sum game has a unique Nash equilibrium in which each player randomizes uniformly among the three strategies. One could conclude that there is not much more to say about the game. But suppose you entered a Rochambeau tournament. Would you simply adopt the equilibrium strategy? If you did, you would not win the competition. This is no idle speculation; such competitions take place routinely. For example, starting in 2002, the World Rock Papers Scissors Society (WRPS) standardized a set of rules for international play and has overseen annual International World Championships as well as many regional and national events throughout the year. These championships have been attended by players from around the world and have attracted widespread international media attention. The winners are never equilibrium players. For example, on October 25th, 2005, 495 people entered the competition in Toronto from countries as diverse as Norway, Northern Ireland, the Cayman Islands, Australia, New Zealand and the UK. The winner was Toronto Lawyer Andrew Bergel, who beat These tournaments of course are not a perfect match with the formal model of repeated games. However, we include the example not only for entertainment value. The rules of the RPS tournaments call for 'matches' between players, each match consisting of several 'games', where a game is a single play of RPS. The early matches adopted a "best of three of three" format, meaning that the player who wins a best of three set garners one point and requires two points to take the match. The Semi-Finals and the Final Match used the "best of three of five" format, meaning that the player who wins a best of three set garners one point and requires three points to take the match. And so the competition really consisted of a series of repeated games, some of them longer than others.  4  Entertainment aside, what do we learn from this? We believe that this is a cautioning tale regarding the predictive or prescriptive role of equilibria in complex games, and in particular in repeated games. There are many examples of games with complex strategy spaces, in which equilibrium analysis plays little or no role-including familiar parlor games, or the Trading Agent Competition (TAC), a computerized trading competition.  5  The strategy space in a repeated game (or more generally a stochastic game) is immense-all mappings from past history to mixed strategies in the stage game. In such complex games it is not reasonable to expect that players contemplate the entire strategy space-their own or that of the opponent(s). Thus, (e.g., Nash) equilibria don't play here as great a predictive or prescriptive role. Our cautioning words should be viewed as countering the default blind adoption of equilibria as the driving concept in complex games, but not as a sweeping statement against the relevance of equilibria in some cases. The simpler the stage game, and the longer its repetition, the more instructive are the equilibria. Indeed, despite our example above, we do believe that if only two players play a repeated RPS game for long enough, they will tend to converge to the equilibrium strategy (this is particularly true of computer programs, that don't share the human difficulty with throwing a mental die). Even in more complex games there are examples where computer calculation of approximate equilibria within a restricted strategy space provided valuable guidance in constructing effective strategies. This includes the game of Poker (  [33]  and  [4] ), and even, as an exception to the general rule we mentioned, one program that competed in the Trading Agent Competition  [13] . Our point has only been that in the context of complex games, so-called "bounded rationality", or the deviation from the ideal behavior of omniscient agents, is not an esoteric phenomenon to be brushed aside.
 A (very partial) sample of MAL work To make the discussion concrete, it is useful to look at MAL work over the years. The selection that follows is representative but very partial; no value judgment or other bias are intended by this selection. The reader familiar with the literature may wish to skip to Section 4.3, where we make some general subjective comments. Unless we indicate otherwise, our examples are drawn from the special case of repeated, two-person games (as opposed to stochastic, n-player games). We do this both for ease of exposition, and because the bulk of the literature indeed focuses on this special case. We divide the coverage into three parts: techniques, results, and commentary.
 Some MAL techniques We will discuss three classes of techniques-one representative of work in game theory, one more typical of work in AI, and one that seems to have drawn equal attention from both communities. entertaining reading; of note is the restriction of the strategy space to Rock, Paper and Scissors, and explicitly ruling out others: "Any use of Dynamite, Bird, Well, Spock, Water, Match, Fire, God, Lightning, Bomb, Texas Longhorn, or other non-sanctioned throws, will result in automatic disqualification". The overview of the RPS society and its tournaments is adapted from the inimitable Wikipedia, the collaborative online encyclopedia, as available on January 2, 2006. Wikipedia goes on to list the champions since 2002; we note without comment that they are all male Torontonians. The results of the specific competition cited are drawn from the online edition of the Boise Weekly dated November 2, 2005. The Boise Weekly starts the piece with "If it weren't true, we wouldn't report on it".  5  http://tac.eecs.umich.edu.
 Model-based approaches The first approach to learning we discuss, which is common in the game theory literature, is the model-based one. It adopts the following general scheme: 1. Start with some model of the opponent's strategy. 2. Compute and play the best response. 3. Observe the opponent's play and update your model of her strategy. 4. Goto step 2. Among the earliest, and probably the best-known, instance of this scheme is fictitious play  [10] . The model is simply a count of the plays by the opponent in the past. The opponent is assumed to be playing a stationary strategy, and the observed frequencies are taken to represent the opponent's mixed strategy. Thus after five repetitions of the Rochambeau game in which the opponent played (R, S, P , R, P ), the current model of her mixed strategy is (R = 0.4, P = 0.4, S = 0.2). There exist many variants of the general scheme, for example those in which one does not play the exact best response in step 2. This is typically accomplished by assigning a probability of playing each pure strategy, assigning the best response the highest probability, but allowing some chance of playing any of the strategies. A number of proposals have been made of different ways to assign these probabilities such as smooth fictitious play  [18]  and exponential fictitious play  [19] . A more sophisticated version of the same scheme is seen in rational learning  [30] . The model is a distribution over the repeated-game strategies. One starts with some prior distribution; for example, in a repeated Rochambeau game, the prior could state that with probability 0.5 the opponent repeatedly plays the equilibrium strategy of the stage game, and, for all k > 1, with probability 2 −k she plays R k times and then reverts to the repeated equilibrium strategy. After each play, the model is updated to be the posterior obtained by Bayesian conditioning of the previous model. For instance, in our example, after the first non-R play of the opponent, the posterior places probability 1 on the repeated equilibrium play.
 Model-free approaches An entirely different approach that has been commonly pursued in the AI literature  [29] , is the model-free one, which avoids building an explicit model of the opponent's strategy. Instead, over time one learns how well one's own various possible actions fare. This work takes place under the general heading of reinforcement learning,  6  and most approaches have their roots in the Bellman equations  [3] . The basic algorithm for solving for the best policy in a known MDP starts by initializing a value function, V 0 : S → R, with a value for each state in the MDP. The value function can then be iteratively updated using the Bellman equation: V k+1 ← R(s) + γ max a s T (s, a, s )V k (s ) The optimal policy can then be obtained by selecting the action, a, at each state, s, that maximizes the expected value: s T (s, a, s )V k (s ). Much of the work in AI has focused strategies for rapid convergence, on very large MDPs, and in particular on unknown and partially observable MDPs. While this is not our focus, we do briefly discuss the unknown case, since this is where the literature leading to many of the current approaches for stochastic games originated. For MDPs with unknown reward and transition functions, the Q-learning algorithm  [53]  can be used to compute an optimal policy. Q(s, a) ← (1 − α t )Q(s, a) + α t R(s, a) + γ V (s ) V (s) ← max a∈A Q(s, a) As is well known, with certain assumptions about the way in which actions are selected at each state over time and constraints on the learning rate schedule, α t , Q-learning can be shown to converge to the optimal value function V * . The Q-learning algorithm can be extended to the multi-agent stochastic game setting by having each agent simply ignore the other agents and pretend that the environment is passive: Q i (s, a i ) ← (1 − α t )Q i (s, a i ) + α t R i (s, a) + γ V i (s ) V i (s) ← max a i ∈A i Q i (s, a i ) Several authors have tested variations of the basic Q-learning algorithm for MAL (e.g.,  [48] ). However, this approach ignores the multi-agent nature of the setting entirely. The Q-values are updated without regard for the actions selected by the other agents. While this can be justified when the opponents' distributions of actions are stationary, it can fail when an opponent may adapt its choice of actions based on the past history of the game. A first step in addressing this problem is to define the Q-values as a function of all the agents' actions: Q i (s, a) ← (1 − α)Q i (s, a) + α R i (s, a) + γ V i (s ) We are however left with the question of how to update V , given the more complex nature of the Q-values. For (by definition, two-player) zero-sum SGs, Littman suggests the minimax-Q learning algorithm, in which V is updated with the minimax of the Q values  [36] : V 1 (s) ← max P 1 ∈Π(A 1 ) min a 2 ∈A 2 a 1 ∈A 1 P 1 (a 1 )Q 1 s, (a 1 , a 2 ) Later work (such as the joint-action learners in  [14]  and the Friend-or-Foe Q algorithm in  [37] ) proposed other update rules for the Q and V functions focusing on the special case of common-payoff (or 'team') games. A stage game is common-payoff if at each outcome all agents receive the same payoff. The payoff is in general different in different outcomes, and thus the agents' problem is that of coordination; indeed these are also called games of pure coordination. The work on zero-sum and common-payoff games continues to be refined and extended (e.g.,  [8, 32, 34, 52] ). Much of this work has concentrated on provably optimal tradeoffs between exploration and exploitation in unknown, zerosum games; this is a fascinating topic, but not germane to our focus. More relevant are the most recent efforts in this line of research to extend the "Bellman heritage" to general-sum games (e.g., Nash-Q by  [25]  and CE-Q by  [21] ). We do not cover these for two reasons: The description is more involved, and the results have been less satisfactory; more on the latter below.
 Regret minimization approaches Our third and final example of prior work in MAL is no-regret learning. It is an interesting example for two reasons. First, it has some unique properties that distinguish it from the work above. Second, both the AI and game theory communities appear to have converged on it independently. The basic idea goes back to early work on how to evaluate the success of learning rules  [5, 23] , and has since been extended and rediscovered numerous times over the years under the names of universal consistency, no-regret learning, and the Bayes envelope (see  [16]  for an overview of this history). We will describe the algorithm proposed in  [24]  as a representative of this body of work. We start by defining the regret, r t i (a j , s i ) of agent i for playing the sequence of actions s i instead of playing action a j , given that the opponents played the sequence s −i . r t i (a j , s i |s −i ) = t k=1 R a j , s k −i − R s k i , s k −i The agent then selects each of its actions with probability proportional to max(r t i (a j , s i ), 0) at each time step t + 1. Recently, these ideas have also been adopted by researchers in the computer science community (e.g.,  [17, 27, 55] ). Note that the application of approaches based on regret minimization has been restricted to the case of repeated games. The difficulties of extending this concept to stochastic games are discussed in  [39] .
 Some typical results One sees at least three kinds of results in the literature regarding the learning algorithms presented above, and others like them. These are: 1. Convergence of the strategy profile to an (e.g., Nash) equilibrium of the stage game in self play (that is, when all agents adopt the learning procedure under consideration). 2. Successful learning of an opponent's strategy (or opponents' strategies). 3. Obtaining payoffs that exceed a specified threshold. Each of these types comes in many flavors; here are some examples. The first type is perhaps the most common in the literature, in both game theory and AI. For example, while fictitious play does not in general converge to a Nash equilibrium of the stage game, the distribution of its play can be shown to converge to an equilibrium in zero-sum games  [46] , 2 × 2 games with generic payoffs  [41] , or games that can be solved by iterated elimination of strictly dominated strategies  [42] . Similarly in AI, in  [38]  minimax-Q learning is proven to converge in the limit to the correct Q-values for any zero-sum game, guaranteeing convergence to a Nash equilibrium in self-play. This result makes the standard assumptions of infinite exploration and the conditions on learning rates used in proofs of convergence for single-agent Q-learning. Claus and Boutilier  [14]  conjecture that both single-agent Q-learners and the belief-based joint action learners they proposed converge to an equilibrium in common payoff games under the conditions of self-play and decreasing exploration, but do not offer a formal proof. Friend-or-Foe Q and Nash-Q were both shown to converge to a Nash equilibrium in a set of games that are a slight generalization on the set of zero-sum and common payoff games. Rational learning exemplifies results of the second type. The convergence shown is to correct beliefs about the opponent's repeated game strategy; thus it follows that, since each agent adopts a best response to their beliefs about the other agent, in the limit the agents will converge to a Nash equilibrium of the repeated game. This is an impressive result, but it is limited by two factors; the convergence depends on a very strong assumption of absolute continuity, and the beliefs converged to are only correct with respect to the aspects of history that are observable given the strategies of the agents. This is an involved topic, and the reader is referred to the literature for more details. The literature on no-regret learning provides an example of the third type of result, and has perhaps been the most explicit about criteria for evaluating learning rules. For example, in  [19]  two criteria are suggested. The first is that the learning rule be 'safe', which is defined as the requirement that the learning rule guarantee at least the minimax payoff of the game. (The minimax payoff is the maximum expected value a player can guarantee against any possible opponent.) The second criterion is that the rule should be 'consistent'. In order to be 'consistent', the learning rule must guarantee that it does at least as well as the best response to the empirical distribution of play when playing against an opponent whose play is governed by independent draws from a fixed distribution. They then define 'universal consistency' as the requirement that a learning rule do at least as well as the best response to the empirical distribution regardless of the actual strategy the opponent is employing (this implies both safety and consistency) and show that a modification of the fictitious play algorithm achieves this requirement. In  [20]  they strengthen their requirement by requiring that the learning rule also adapt to simple patterns in the play of its opponent. The requirement of 'universal consistency' is in fact equivalent to requiring that an algorithm exhibit no-regret, generally defined as follows, against all opponents. ∀ > 0, lim t→inf 1 t max a j ∈A i r t i (a j , s i |s −i ) < In both game theory and artificial intelligence, a large number of algorithms have been show to satisfy universal consistency or no-regret requirements. In addition, recent work  [6]  has tried to combine these criteria resulting in GIGA-WoLF, a no-regret algorithm that provably achieves convergence to a Nash equilibrium in self-play for games with two players and two actions per player. Meanwhile, the regret matching algorithm  [24]  described earlier guarantees that the empirical distributions of play converge to the set of correlated equilibria of the game. Other recent work by  [2]  has addressed concerns with only requiring guarantees about the behavior in the limit. Their algorithm is guaranteed to achieve -no-regret payoff guarantees with small polynomial bounds on time and uses only the agent's ability to observe what payoff it receives for each action.
 Some observations and questions We have so far described the work without comment; here we take a step back and ask some questions about this representative work. Our first comment concerns the settings in which the results are presented. While the learning procedures apply broadly, the results for the most part focus on self play (that is, when all agents adopt the learning procedure under consideration). They also tend to focus on games with only two agents. Why does most of the work have this particular focus? Is it technical convenience, or is learning among more than two agents, or among agents using different learning procedures, less relevant for some reason? Our second comment pertains to the nature of the results. With the exception of the work on no-regret learning, the results we described investigate convergence to equilibrium play of the stage game (albeit with various twists). Is this the pertinent yardstick? If the process (say of self play between two agents) does not converge to equilibrium play, should we be disturbed? More generally, and again with the exception of no-regret learning, the work focuses on the play to which the agents converge, not on the payoffs they obtain. Which is the right focus? No-regret learning is distinguished by its starting with criteria for successful learning, rather than a learning procedure. The question one might ask is whether the particular criteria are adequate. In particular, the requirement of consistency ignores the basic lesson regarding learning-vs.-teaching discussed in Section 3. By measuring the performance only against stationary opponents, we do not allow for the possibility of teaching opponents. Thus, for example, in an infinitely repeated Prisoners' Dilemma game, no-regret dictates the strategy of always defecting, precluding the possibility of cooperation (for example, by the mutually reinforcing Tit-For-Tat strategies). Our goal here is not to critique the existing work, but rather to shine a spotlight on the assumptions that have been made, and ask some questions that get at the basic issues addressed, questions which we feel have not been discussed as clearly and as explicitly as they deserve. In the next section we propose an organized way of thinking about these questions.
 Five distinct agendas in multi-agent learning After examining the MAL literature-the work surveyed here and much else-we have reached the conclusion that there are several distinct agendas at play, which are often left implicit and conflated. We believe that a prerequisite for success in the field is to be very explicit about the problem being addressed. We ourselves can identify five distinct possible goals of MAL research. There may well be others, but these are the ones we can identify. They each have a clear motivation and a success criterion that will allow researchers to evaluate new contributions, even if people's judgments may diverge regarding their relative importance or success to date. They can be caricatured as follows: 1. Computational 2. Descriptive 3. Normative 4. Prescriptive, cooperative 5. Prescriptive, non-cooperative We can now consider each of the five in turn. The first agenda is computational in nature. It views learning algorithms as an iterative way to compute properties of the game, such as solution concepts. As an example, fictitious play was originally proposed as a way of computing a sample Nash equilibrium for zero-sum games  [10] , and replicator dynamics has been proposed for computing a sample Nash equilibrium in symmetric games. Other adaptive procedures have been proposed more recently for computing other solution concepts (for example, computing equilibria in local-effect games  [35] ). These tend not to be the most efficient computation methods, but they do sometimes constitute quick-and-dirty methods that can easily be understood and implemented. The second agenda is descriptive-it asks how natural agents learn in the context of other learners. The goal here is to investigate formal models of learning that agree with people's behavior (typically, in laboratory experiments), or possibly with the behaviors of other agents (for example, animals or organizations). This same agenda could also be taken to apply to large-population models, if those are indeed interpreted as representing populations. This problem is clearly an important one, and when taken seriously calls for strong justification of the learning dynamics being studied. One approach is to apply the experimental methodology of the social sciences. There are several good examples of this approach in economics and game theory, for example  [15]  and  [11] . There could be other supports for studying a given learning process. For example, to the extent that one accepts the Bayesian model as at least an idealized model of human decision making, one could justify Kalai and Lehrer's model of rational learning.  7  However, it seems to us that sometimes there is a rush to investigate the convergence properties, motivated by the wish to anchor the central notion of game theory in some process, at the expense of motivating that process rigorously.  8  The centrality of equilibria in game theory underlies the third agenda we identify in MAL, which for lack of a better term we called normative, and which focuses on determining which sets of learning rules are in equilibrium with each other. More precisely, we ask which repeated-game strategies are in equilibrium; it just so happens that in repeated games, most strategies embody a learning rule of some sort. For example, we can ask whether fictitious play and Q-learning, appropriately initialized, are in equilibrium with each other in a repeated Prisoner's Dilemma game. Although one might expect that game theory purists might flock to this approach, there are very few examples of it. In fact, the only example we know originates in AI rather than game theory  [9] , and it is explicitly rejected by at least some game theorists  [18] . We consider it a legitimate normative theory. Its practicality depends on the complexity of the stage game being played and the length of play; in this connection see our discussion of the problematic role of equilibria in Section 3. The last two agendas are prescriptive; they ask how agents should learn. The first of these involves distributed control in dynamic systems. There is sometimes a need or desire to decentralize the control of a system operating in a dynamic environment, and in this case the local controllers must adapt to each other's choices. This direction, which is most naturally modeled as a repeated or stochastic common-payoff (or 'team') game, has attracted much attention in AI in recent years. Proposed approaches can be evaluated based on the value achieved by the joint policy and the resources required, whether in terms of computation, communication, or time required to learn the policy. In this case there is rarely a role for equilibrium analysis; the agents have no freedom to deviate from the prescribed algorithm. Examples of this work include  [12, 14, 22]  to name a small sample. Researchers interested in this agenda have access to a large body of existing work both within AI and other fields such as control theory and distributed computing. In our final agenda, termed 'prescriptive, non-cooperative', we ask how an agent should act to obtain high reward in the repeated (and more generally, stochastic) game. It thus retains the design stance of AI, asking how to design an optimal (or at least effective) agent for a given environment. It just so happens that this environment is characterized by the types of agents inhabiting it, agents who may do some learning of their own. The objective of this agenda is to identify effective strategies for environments of interest. An effective strategy is one that achieves a high reward in its environment, where one of the main characteristics of this environment is the selected class of possible opponents. This class of opponents should itself be motivated as being reasonable and containing opponents of interest. Convergence to an equilibrium is not a goal in and of itself. There are various possible instantiations of the term 'high reward'. One example is the no-regret line of work which we discussed. It clearly defines what it means for a reward to be high enough (namely, to exhibit no regret); we also discussed the limitations of this criterion. A more recent example, this one from AI, is  [7] . This work puts forward two criteria for any learning algorithm in a multi-agent setting:  (1)  The learning should always converge to a stationary policy, and (2) if the opponent converges to a stationary policy, the algorithm must converge to a best response. There are possible critiques of these precise criteria. They can be too weak since in many cases of interest the opponents will not converge on a stationary strategy. And they can be too strong since attaining a precise best response, without a constraint on the opponent's strategy, is not feasible. But this work, to our knowledge, marks the first time a formal criterion was put forward in AI. A third example of the last agenda is our own work in recent years. In  [45]  we define a criterion parameterized by a class of 'target opponents'; with this parameter we make three requirements of any learning algorithm: (1) (Targeted optimality) The algorithm must achieve an -optimal payoff against any 'target opponent', (2) (Safety) The algorithm must achieve at least the payoff of the security level strategy minus against any other opponent, and (3) (Autocompatibility) The algorithm must perform well in self-play (the precise technical condition is omitted here). We then demonstrate an algorithm that provably meets these criteria when the target set is the set of stationary opponents in general-sum two-player repeated games. More recent work has extended these results to handle opponents whose play is conditional on the recent history of the game  [44]  and settings with more than two players  [51] .
 Summary In this article we have made the following points: 1. Learning in MAS is conceptually, not only technically, challenging. 2. One needs to be crystal clear about the problem being addressed and the associated evaluation criteria. 3. For the field to advance one cannot simply define arbitrary learning strategies, and analyze whether the resulting dynamics converge in certain cases to a Nash equilibrium or some other solution concept of the stage game. This in and of itself is not well motivated. 4. We have identified five coherent agendas. 5. Not all work in the field falls into one of these buckets. This means that either we need more buckets, or some work needs to be revisited or reconstructed so as to be well grounded. There is one last point we would like to make, which didn't have a natural home in the previous sections, but which in our view is important. It regards evaluation methodology. We have focused throughout the article on formal criteria, and indeed believe these to be essential. However, as is well known in computer science, many algorithms that meet formal criteria fail in practice, and vice versa. And so we advocate complementing the formal evaluation with an experimental one. We ourselves have always included a comprehensive bake-off between our proposed algorithms and the other leading contenders across a broad range of games. The algorithms we coded ourselves; the games were drawn from GAMUT, an existing testbed (see  [43]  and http://gamut.stanford.edu). GAMUT is available to the community at large. It would be useful to have a learning-algorithm repository as well. To conclude, we re-emphasize the statement made at the beginning: This article is meant to be the beginning of a discussion in the field, not its end. Fig. 1 .Fig. 2 . 12 Fig. 1. Stackelberg stage game: The payoff for the row player is given first in each cell, with the payoff for the column player following.
			 We acknowledge a simplification of history here. There is definitely MAL work in AI that predates the last few years, though the relative deluge is indeed recent. Similarly, we focus on AI since this is where most of the action is these days, but there are also other areas in computer science that feature MAL material; we mean to include that literature here as well.
			 We do acknowledge some degree of humor in the example. The detailed rules in http://www.rpschamps.com/rules.html make for additional
			 We note that the term is used somewhat differently in the game theory literature.
			 Although this is beyond the scope of this article, we note that the question of whether one can justify the Bayesian approach in an interactive setting goes beyond the familiar contravening experimental data; even the axiomatic justification of the expected-utility approach does not extend naturally to the multi-agent case. 8  It has been noted that game theory is somewhat unusual in having the notion of an equilibrium without associated dynamics that give rise to the equilibrium [1] .