A Cultural Decision-Making Model for Negotiation based on Inverse Reinforcement Learning

A Cultural Decision-Making Model for Negotiation based on Inverse Reinforcement Learning Elnaz Nouri (nouri@ict.usc.edu) Kallirroi Georgila (kgeorgila@ict.usc.edu) David Traum (traum@ict.usc.edu) Institute for Creative Technologies, University of Southern California 12015 Waterfront Drive, Playa Vista, CA 90094, USA Abstract We learn culture-specific weights for a multi-attribute model of decision-making in negotiation, using Inverse Reinforcement Learning (IRL). The model takes into account multiple indi- vidual and social factors for evaluating the available choices in a decision set, and attempts to account for observed be- havior differences across cultures by the different weights that members of those cultures place on each factor. We apply this model to the Ultimatum Game and show that weights learned from IRL surpass both a simple baseline with random weights, and a high baseline considering only one factor of maximiz- ing gain in own wealth in accounting for the behavior of hu- man players from four different cultures. We also show that the weights learned with our model for one culture outperform weights learned for other cultures when playing against oppo- nents of the first culture. We conclude that decision-making in negotiation is a complex, culture-specific process that can- not be explained just by the notion of maximizing one’s own utility, but which can be learned using IRL techniques. Keywords: cultural decision-making; negotiation; ultimatum game; inverse reinforcement learning. Introduction Social scientists have often observed that people from dif- ferent cultures behave differently in interactive situations (Camerer, 2003; Roth, Prasnikar, Okuno-Fujiwara, & Zamir, 1991). There are several different possible explanations for this, including 1. one culture is better than another at optimizing outcomes; 2. there is some kind of convention (Lewis, 1969) or equi- librium at work, such that people behave differently be- cause the context is different, particularly their expecta- tions about how others will behave. For example people in Japan or England drive on the left while people from America and Europe drive on the right, because that is the safest, most efficient way given how other drivers will be- have, even though the goals of safety and efficiency are the same, and neither is innately better at achieving these goals; 3. the cultures have different goals, which lead to their opti- mizing different functions. Most classical economic game-theory accounts of decision-making, e.g. (Neumann & Morgenstern, 1944), look at a monolithic notion of utility and maximizing expected utility as the key to rationality. This, in effect, denies the third explanation above. For very simple games, where it is relatively easy to calculate the payoffs, the first possibility seems hard to believe, thus we are left with the hypothesis that differences in behavior are based on applying common utility principles to different problems. Others, e.g. (Gal, Pfeffer, Marzo, & Grosz, 2004), have claimed that there are many factors that contribute to the behavior of humans in social situations. This makes the third explanation plausible, if people from different cultures have different relative weights for the different factors. But this leads to a further question of how to determine those different weights. In (Nouri & Traum, 2011) we presented one such model of decision-making that culture-specific virtual agents were able to use to play the Ultimatum Game (see the following section) with each other or with people. The model used Hofstede’s multi-dimensional model of culture (Hofstede, 2001) to determine the relative weights of different factors. However, in that work the weights were set manually using our intuitions about how to apply the literature, which involved a number of relatively arbitrary decisions. In this paper we attempt to learn the weights using In- verse Reinforcement Learning (IRL) (Abbeel & Ng, 2004). To our knowledge no one has used IRL before in the Ulti- matum Game or generally to learn patterns of behavior in negotiation. We also perform two experiments to try to get at the question above of what is the best explanation for the observed behavioral differences across cultures. On one ac- count, it is the different goals that lead to different behavior. In this case we would predict that we learn different goals for different cultural patterns and that these goals would be bet- ter at generating observed behavior than other possible goals. On another account, we would expect the same set of goals to be satisfactory for any population, and differences in behav- ior to result from the different environments that are encoun- tered. Our results show that the learned weights are better able to match observed distributions of culture-specific be- havior than either arbitrary weights, a simple model based on economic gain, or in most cases the weights learned for other cultures. This suggests that cultures vary in goals, not just conventional circumstances but also that we can successfully use IRL techniques to learn population-specific goals for this type of game.