Opponent Models and Heuristic Strategies for Simple Games

Opponent Models and Heuristic Strategies for Simple Games Timothy J. Vickery (vickery@wjh.harvard.edu) Department of Psychology, 33 Kirkland St. Cambridge, MA 02138 joint decisions. Nash (1950) determined that any such scenario has at least one solution strategy from which no player can benefit by deviating. This solution, called Nash equilibrium, is an optimal solution to a game when all agents behave “rationally,” in terms of decision theory. The games typically submitted to game theoretic analyses are analytically tractable: far simpler than games like chess, but useful for probing the capabilities of agents. Two games employed by Stahl and Wilson (1995) to study human behavior are shown in table 1. The model presented in this paper will later be applied to the data they collected as humans played these games, which are fairly prototypical of situations studied in game theory. In these games, each subject is asked to examine each matrix, in turn, and choose T, M or B as a response. The player’s reward is determined by values in the chosen row. Each cell represents the value of an individual’s choice given the choices of her opponent, which correspond to the columns. Thus, if a player chose T and her opponent chose “M,” she would receive the payoff in cell (T,“M”). The games used here are symmetric, meaning that the opponent’s payoff table is the same. The prescriptive qualities of game theoretic models are fascinating and useful as benchmarks to the success of algorithms in approximating solutions, but behavioral economists and psychologists have found that the traditional forms are unfortunately dysfunctional as descriptive models in many situations (cf. Camerer, 2003, who reviews many behavioral observations that disconfirm predictions relying on the concept of Nash equilibrium). The most successful models of human behavior in repeated games depict humans as employing algorithms that depend on reward history to approximate an optimal solution (for example, reinforcement algorithms). In general, humans do not seem to employ strategies that correspond to Nash equilibrium strategies either initially or over repeated interactions, but rather they often approximate the Nash equilibrium strategy in a manner consistent with reinforcement learning or similar algorithms that depend on historical factors. Behavior matching the Nash equilibrium in one-shot (non-repeated) games would imply either infinitely recursive reasoning or explicit knowledge of how to calculate Nash equilibrium. Thus, there is little reason to believe that humans bring to a game any mechanism tuned to extracting an optimal strategy like the Nash equilibrium solution without any experience at all with a particular incentive structure. The behavioral data reported by Stahl and Wilson (1995) are testament to this: the Nash equilibrium strategy alone is entirely inadequate for explaining behavior. However, the authors presented a successful model of behavior in one-shot games, in which some players were modeled as having a concept of others Abstract This study introduces a model that describes the reasoning strategies of a population of players in simultaneous-move, one-shot games. In the past, models of such behavior have explicitly employed the concept of Nash equilibrium in players’ models of other players. In this model, behavior is accounted for in terms of simple heuristic strategies and opponent modeling, rather than by recourse to the concept of Nash equilibrium. The model represents six types of boundedly rational players: three types who employ no model of their opponents, two types who model their opponents as employing simple heuristics, and one type who models the population as a mixture of different types of players. Results show promise for eliminating the concept of Nash equilibrium from players’ models of other players. In pursuit of this goal, a new type of graphical model based on Influence Diagrams is developed. Uncertain Decision Diagrams are suitable for modeling human decision-making with respect to explicit mental models that include noisy estimates of utility, and can be extended to model players’ models of other players. Introduction Agents often confront situations in which not only their personal choices, but also the choices of others, affect the subjective value of an outcome. Some examples of these situations are the negotiation of hunting rights between two tribes, the barter of goods at a market, and a game of rock, paper, or scissors. Such problems are of great interest to cognitive science, because humans evolved and exist in an environment replete with demands to compete and opportunities for cooperation. An understanding of how humans reason about competitive scenarios would improve our understanding of human behavior in numerous ways, allowing us to probe questions about the representation of utility, other agents, and the world, and enabling the construction of artificial agents that exploit or enhance the strengths and weaknesses of human decision making. The model sketched here describes how humans make decisions in games, or formalized incentive structures involving multiple agents. We model reasoning strategies employed when a person has no reward history with a game (i.e., the games are one-shot, not repeated). The model describes a population of players as a mixture of player types, some of whom employ simple heuristic strategies, and others of whom employ simple models of opponents who play by these heuristic strategies. Game Theory Background Traditional game theory formalizes the basic problem of determining the optimal solution for all agents in a setting in which outcomes for agents are determined by