LEARNING FROM OBSERVATION FOR INTERACTIVE COMPUTER GAMES

The research presented in this paper describes an architecture, which enables an agent to predict an observed entity’s actions (most likely a human’s) online. Case-based approaches have been utilised by a number of researchers for online action prediction in interactive applications. Our architecture builds on these works and provides a number of novel contributions. Specifically our architecture offers a more comprehensive state representation, behaviour prediction and a more robust case maintenance approach. The proposed architecture is fully described in terms of interactive simulations (specifically first person shooter (FPS) computer games); however it would be applicable to other interactive applications, such as intelligent tutoring and surveillance systems. We conclude the paper by evaluating our proposed architecture and discussing how the system will be implemented. INTRODUCTION & MOTIVATION Action games, in particular first person shooters (FPS) are one of the most popular game genres on the market today (Bauckhage and Thurau, 2004). These games usually involve running around and using deadly force against an enemy (Laird and Lent, 2001). Examples include Half Life, Quake and Unreal. AI opponents in actions games are becoming more sophisticated and are beginning to approximate the game play of human players (Laird and Lent, 2001); however non-player characters (NPCs) (agents) in these types of games tend to rely on pre-programmed scripts and finite state machines to achieve this behaviour. In fact almost all commercial computer game AI (regardless of genre) is controlled by scripts or finite state machines (Cass, 2002). The use of these techniques can result in poor AI, which is predicable, less believable and can be exploited by the human player (Spronck et al. 2003). This reduces the enjoyment for the player and can result in them preferring human controlled opponents (Schaeffer, 2001). Unfortunately human opponents are not always available or appropriate, so continuing to improve computer game AI is a desirable endeavour. In addition with the on going developments in computer hardware, more processing time (Cass, 2002) is now available for a computer game’s AI. In the future games developers will be able to employ more sophisticated academic AI techniques, to make the NPCs in their computer games more human-like and responsive to the game player. Learning from observation is one such sophisticated AI approach. Traditionally human behaviour models have been developed by acquiring knowledge from subject matter experts (SMEs) (Fernlund, 2002). This is a difficult and time-consuming process, which learning from observation attempts to automate. In AI research there is no hard and fast definition for the term “learning from observation”. Most literature refers to it as a method of learning the behaviour of another entity by observing its actions (Fernlund, 2002). This is a broad description and means that learning from observation can be applied to many types of applications / problems, using a variety of learning techniques. In the area of computer games there have been a number of learning from observation research efforts. Lent and Laird implemented their symbolic KnoMic system with the Soar architecture, in an air combat domain and have created a reactive Quake II bot (Laird and Lent, 2001). Games researchers have also examined sub-symbolic approaches to the problem. One of the most interesting is Thurau et al. (2004) system, which makes use of artificial neural network (ANN) techniques to model human behaviour in order to create a reactive NPC for Quake II. This system learns offline from recorded Quake II game sessions, between human players. McGlinchey (2003) also uses a similar approach to create more humanlike opponents for the Pong game. A system called GoCaps (Game observation capture) (Alexander, 2002) also learns by observing play sessions between humans, but rather than from recorded game sessions the system learns in a training mode and is directly controlled by a human player / expert. All these approaches produce believable behaviour for NPCs through learning from observation. However, these types of systems still encounter the same limitations as current FSM and scripting techniques. This is highlighted by McGlinchey (2003) who states that humans may act differently in identical situations, but his system is always completely deterministic. In addition Laird found that games developers were impressed by his behaviour modelling system, but would invariably ask, “Does it anticipate the human player’s actions? ” (Laird, 2001). Enabling an agent to predict and adapt its behaviour through learning from observation (I.e. rather that simply using observations to learn, we want to model another entity’s behaviour in order to predict and adapt) would overcome the limitations facing traditional and some sophisticated computer game AI approaches. As a result the primary goal of our research is to explore and evaluate how learning from observation can be used to develop agents for interactive applications (specifically FPS computer games), which are able to intelligently adapt their behaviour in order to improve their chances of success in cooperative or adversarial situations. We begin this paper by defining the computer game scenario that will form the focus of our research. After that we discuss background and related work. We then take the first steps to achieving our goal by developing an architecture inspired by existing case-based prediction research and proposing a number of novel contributions which improve these types of systems. Finally we conclude the paper by evaluating our architecture and discussing how the proposed system will be implemented. COMPUTER GAME SCENARIO The test-bed for our adaptation research is FPS computer games. We will initially focus our work on applying action and behaviour adaptation to combat situations in these types of games (as illustrated in Figure 1); however future work will incorporate adaptation at task / tactic and goal levels. When an NPC is not in a combat situation it executes longterm plans, goals and certain tactics, such as avoiding combat in order conserve health (Figure 1 (a)). High-level goals include searching for items (i.e. weapons), navigating to a location, searching for opponents or capturing a flag. When the NPC enters an area near where an opponent is located, Figure 1 (b), the NPC in effect enters a combat situation and has to decide quickly in real time what actions, behaviours and tactics to take. Actions can be considered atomic events (e.g. move left, move forward). Behaviours can be classed as identifiable components or building blocks (Khoo et al. 2003) of a tactic / task. For example the ‘Camp’ tactic could consist of 4 behaviours, find hidden location, select weapon, wait for target and shoot target (Lent and Laird, 1999), some of which could have sub-behaviours. Also it’s worth noting that behaviours can be classified in a number of other ways, such as stimulus-responses (i.e. reactive behaviours), however this is not the context in which they are used in this paper. Tactics can be described as smart, localized situation handling, which depend on anticipating an enemy and a broad understanding of a scene (e.g. the layout of the environment) (Bauckhage and Thurau, 2004). Examples of tactics include strafing, camping and sniping. It would be desirable for an NPC in this type of situation to be able to adapt and reason about an opponent’s actions, behaviours and tactics in order to improve its decision making and chances of survival. For example predicting an opponent a few time-steps into the future could allow an NPC to adapt its path planning or targeting, by providing possible future locations to aim for. This would be especially useful in chase situations, where an NPC has to catch or kill an opponent. Inferring an opponent’s tactics would allow an NPC to adapt its playing style, for example if an opponent prefers to snipe, the NPC could adapt its behaviour to prefer cautious tactics and to search for sniper locations. Any approach would however have to operate in real time, as humans do, in order to provide useful adaptation. BACKGROUND & RELATED RESEARCH In order for our proposed system to adapt successfully at different levels of behaviour (i.e. goal, task or action) and be able to integrate with existing frameworks we will make use of a multi-layered adaptation approach. In general, system decision models can be categorised / organised into a hierarchy that follows human behaviour models, such as the psychological hierarchy of human behaviour (Hollnagel, 1994 in Thurau et al. 2004) or cognitive models (Funge et al. 1999 in Dinerstein and Egbert 2005). Kerkez and Cox (2001) define prediction in terms of local (predicting the agent’s immediate course of action) and global (predicting an agent’s goals and plans). Dinerstein and Egbert (2005) make use of a layered behaviour model, which has goal selection on the top, task selection on the second level and action selection on the bottom. For our research we will follow this style of behaviour model, however initial work will focus on action and behaviour prediction. Formulating and integrating other layers will be discussed in future work; however each layer in our proposed system will offer different levels of adaptation, using appropriate learning techniques, which reflect the temporal granularity of the level. The research described in this paper expands on existing work in the area of intelligent online behaviour adaptation. Work in this field has made use of a variety of approaches, such as prediction (Dinerstein et al. 2005), user modelling (Schaeffer et al. 1999), anticipation (Laird, 2001), reinforcement learning (Spronck et al. 2003) and plan recognition (Kerkez and Cox, 2003). We propose to achieve action and behaviour adaptation through prediction and plan recog