JAIST Repository
https://dspace.jaist.ac.jp/Title

少数の記録からプレイヤの価値観を機械学習するチー
ムプレイAIの構成

Author(s)和田, 尭之CitationIssue Date

2015‑03

TypeThesis or Dissertation

Text version

author

URL

http://hdl.handle.net/10119/12785

RightsDescription

Supervisor:池田心, 情報科学研究科, 修士

Japan Advanced Institute of Science and Technology

Design of a Teammate AI by Learning Human-player Utility froma few Records of Actions
Wada Takayuki(1310082)
School of Information Science,
Japan Advanced Institute of Science and Technology
February 12, 2015

Keywords: Game AI, RPG, Utility, Machine Learning, Team-mate, CooperationGame.

One of the goals of game AI research is to create computer players (hereinafter referredto as game AIs) able to entertain human players. Until now, most researchers in thatarea tried to entertain human players by creating suﬃciently strong AIs. As a result,game AIs are strong enough to compete with professional players in Othello, chess or
shogi. Additionally, creating game Ais in many popular computer games is also a subjectof study, like FPS games or RTS games. Methods to create game AIs that are strong orthat behave naturally have been proposed. However, there are few researches for gameAIs to entertain human players as their teammates. Some genres of commercial computer
games, especially RPG games, often allow players to play the game with the game AIs asthe teammates. To entertain human players in these games, it is important to make theaction patterns of the teammate AIs more sophisticated than the opponent AIs. These
teammate AIs are required to cooperate with human players but the AIs often take actions
that human players do not expect them to do. Such mismatches between the expectationsof the human players and the actions taken by the AI players often cause dissatisfaction ofthe players. One of the reasons for such mismatches is that there are several types of subgoals in these games, like “win as soon as possible” or “win with as little received damage
as possible”, and the AI players act without understanding which types of sub-goals areimportant for each human player. The purpose of this study is to propose a method todevelop teammate AI players that estimate the sub-goal preference of the human players
and act with causing less dissatisfaction of the players. First of all, we designed andimplemented a RPG game with rules suitable for this study. We picked out the designof role-playing video games famous recently in Japan, in which players select actions inc 2015 by Wada Takayuki
Copyright ⃝1their turn, from among several types of RPG games. The situations in the game areunfair and mostly advantageous for human players. However, if the player wins, the laststate of the game is sometimes preferable or unpreferable (e.g. some of the teammates arebadly injured or no “Magic power” remains) for the players. These preferences for somegame states are not expressed in numerical form. These game features are not rare insuch games and are also included in our implemented game. In this study, we constructa model of such human preference with a function. The input of the model functionis the game state and the selected actions, and the output is a number quantifying thepreference. By adjusting the variable parameters in this model function, we mimic thepreference for the game states of each human player to some extent. Additionally, weconstruct AI players that select actions using the adjusted model function. These AIplayers have the same preference for the game states as the human players and act with
causing less dissatisfaction of the players. To estimate the human’s preference, we use
Monte-Carlo simulations and simulate how the game state will be at the end of the gameby the human player’s selected action. The performance of Monte-Carlo simulations often
gets improved by selecting more frequently plausible actions. Although, the plausibleactions by human players vary according to the sub-goals that the players have. Thus
we defined a “strategy” (a strategy makes the action patterns of the player biased in
the random simulations) and simulated the game states applying some strategies to theplayers in the random simulations. This approach succeeded in improving the accuracy ofthe estimation of the human player’s preferences. We did an evaluation experiment withartificial human players that select their actions by fixed value functions and we tried toestimate the preferences of these artificial human players by our proposed method. The
selected actions based on the estimated preferences were the same as the selected actions
by the original artificial players at the rate of 67.1% in one setting. The upper bound ofthe rate is about 70.6%(in this setting), which is the rate at which the same actions are
selected when the preferences are exactly the same. Thus the proposed method is only
3.5% inferior in performance in the worst case compared to an ideal estimation. Moreover,
we did an evaluation experiment with human players and tried to estimate the preferenceof the players. At first, we gave them orders, like “fight trying to save Magic Powers of thecharacters” or “try to win the game as soon as possible” before the games. We estimatedthe preferences of the players through 4 games. Secondly, each human player played thegame with a game AI as a teammate. There are two types of these teammate AIs, that is,
AIs selecting their actions by the estimated value functions, or some fixed value functions
that is unrelated to the human’s preference. The human likeness of their behavior isranked on a five grade scale. The proposed method got higher grade (0.7 on average),thus, our proposed method succeeded in estimating the human players’ preference for the2game states and acted without giving the human players uncomfortable feelings.

3