Hierarchical Parallel Markov Models of Interaction

Finite state techniques are popular in entertainment software production, but they complicate the modeling of certain aspects of social engagement. In this paper we examine the problem of building probabilistic finite-state interaction models that allow both hierarchical composition of behaviors, and their parallel engagement. Finally, we propose an extension that resolves the difficulties for a class of common cases. Interaction Modeling Intelligent game agents must be able to interact with players, to present themselves as actual inhabitants of the virtual world. Popular approaches to interaction, such as static conversation trees and exchanges of pre-recorded blurbs, are widely recognized as insufficient, but a more generative natural language interaction will require a different approach to the underlying representation. Interaction modeling in games is constrained by the demands of authoring, believability, and performance. The underlying mechanism must allow for easy behavior authoring, because developers require precise control over what the system will do at runtime, in order to provide enjoyable and engaging user experience, with specific aesthetic effects (Hunicke, LeBlanc and Zubek 2004). Additionally, developers want the behavior of the system to be believable, resembling the relevant aspects of human activity; ideally, believable behavior should be produced out of patterns matched against the game state, rather than being completely hand-scripted by a writer. Finally, the mechanisms must be computationally efficient, producing desirable behavior in only a fraction of total compute cycles. For these reasons, simple techniques such as finite state machines (FSMs) remain very popular. Hand-crafted FSMs are routinely used to implement character behavior, since they are simple to author, intuitive to debug, and efficient to compute. In character interaction and conversation, simple dialog trees are very popular. In domains where the inputs are noisy or ambiguous— such as in tracking human activity or language processing—stochastic generalizations are often used instead. Hidden Markov models (HMMs), for example, improve robustness by allowing for uncertainty; they continually re-estimate the probability of being in a given state, based on observed inputs. The Hierarchical Parallel Problem Rich, believable communication requires engagement in different social protocols, and broad competence in human conversational moves. Implementing the system using a large dialog network is clearly problematic. First, it is an authoring nightmare: finite-state models with non-trivial numbers of spaces are difficult to design and debug. Second, probabilistic implementations become computationally expensive, because HMM belief computation is superlinear in the number of states. We can reduce the complexity of the system by decomposing the FSM into smaller, coupled FSMs. Consider a typical fantasy role-playing game. When building a shopkeeper agent, for example, we can decompose the selling interaction into the threads of deciding on an item, evaluating the item, haggling about the price, and so on; the evaluation thread can consist of praising the workmanship, pointing out features, or defusing criticism; praising the workmanship can then finally bottom out in particular speech acts. The decomposed system would be more efficient. And since individual modules can be developed independently and reused, it should also be cheaper to develop. A natural implementation of such decomposition would be as an augmented transition network (ATN) or pushdown automaton (PDA), which are essentially FSMs that can “call” other FSMs that later “return” to the original FSM. However, in real communication, multiple interactions can be interleaved concurrently. For example, consider the barter example above. Barter includes the child protocols of haggling and praising, simplified versions of which are shown as the linear chains H and P in Figure 1. In practice, these protocols are often, but not always, interleaved, as in: P1 H1 H2 P2 P3 P4 H3 H4. To follow the progress of the barter, the system must therefore be able to accept arbitrary interleavings of these sequences without having to merge them into one (very large) FSM containing separate paths for every possible interleaving. We therefore want our system to support not only hierarchical representations of social protocols, but parallel instantiations of them. Unfortunately, PDAs do not support concurrent execution. In the remainder of this paper, we will show how parallel and hierarchical structure can be modeled using a set of parallel, coupled, Markov models (essentially probabilistic FSMs). This allows flexible, robust interaction using a small number of computationally efficient, easily authored finite-state models. HMM Essentials and Extensions We implement coupling through extensions to the hidden Markov model (HMM) estimation. HMMs are effectively FSMs that keep track of a probability distribution over all possible states rather than a single “current state”. While somewhat more expensive than FSMs, HMMs are much more robust when the input is insufficient to unambiguously determine the current state of the system. The formal definition is as follows. Let  = [0, 1] be the subset of  representing valid probability values. A hidden Markov model M is a tuple ‹ S, A, p, q ›, where: · S = { s1, s2, . } is the set of possible states, with a unique initial state s1, · A = { γ1, γ2, . } is the set of discrete communicative actions (speech acts, etc.) that can be observed, · p : S × S →  gives the probability of transitioning from one state to another, and · q : S × A →  gives the probability of observing a given action when the system is in a given state. To indicate the semantics of the probability functions, we will write q (γ | s) as the probability of action γ being observed at state s, and p (s’ | s) as probability of transition to s’ from s, where ∀s’ ∈ S: ∑ ∈S s p (s’ | s) = 1. Note that this defines action observation as dependent on state, rather than state transitions. These two representations are equally valid and easily convertible, using methods such as those described by Jelinek (1997). For our purpose, however, state-based observation will simplify the description. Estimating the Current State The precise “state” of interaction is not observable at any given time. However, an HMM allows us to estimate it based on a history of observations. The belief distribution over all states is a function b : S ×  → , written as bt (s), which is the system’s estimate of the probability of the process being in state s at time t. The belief can be computed using popular forward algorithm for hidden Markov models (Jelinek 1997, Jurafsky and Martin 2000) extended to the state-based observation model: bt (s) = ∑ ∈S si p (s | si) q (γt | s) bt-1 (si) η (1) Here η is a normalization value use to force bt to sum to 1. Furthermore, for the initial time slice t = 1, define bt (s1) = 1, and bt (s) = 0 for all s ≠ s1. We will not discuss how to interpret the belief distribution function here, but rather jump right into the extensions—the interested reader is encouraged to reference Zubek (2004) or Roy et al. (2000) for more details. When the system has to choose actions based on belief state, then the HMM is called a partially observable Markov decision problem or “POMDP”. We then assume there is a policy π : (S → ) → A that maps belief states to actions.