Eurographics/ ACM SIGGRAPH Symposium on Computer Animation (2008)
M. Gross and D. James (Editors)

Puppet Master: Designing Reactive Character Behavior by
Demonstration
James E. Young1,3 and Takeo Igarashi2,3 and Ehud Sharlin1
1 University of Calgary 2 The University of Tokyo 3 JST ERATO

Abstract
Puppet Master is a system that enables designers to rapidly create interactive and autonomous animated character behaviors that react to a main character controlled by an end-user. The behavior is designed by demonstration,
allowing non-technical artists to intuitively design the style, personality, and emotion of the character, traits which
are very difficult to design using conventional programming. During training, designers demonstrate paired behavior between the main and reacting characters. During run time, the end user controls the main character andthe system synthesizes the motion of the reacting character using the given training data. The algorithm is anextension of Image Analogies [HJO∗ 01], modified to synthesize dynamic character behavior instead of an image.
We introduce non-trivial extensions to the algorithm such as our selection of features, dynamic balancing between
similarity metrics, and separate treatment of path trajectory and high-frequency motion texture. We implemented
a prototype system using physical pucks tracked by a motion-capture system and conducted a user study demonstrating that novice users can easily and successfully design character personality and emotion using our systemand that the resulting behaviors are meaningful and engaging.
Categories and Subject Descriptors (according to ACM CCS): I.3.6 [Computer Graphics]: Methodology andTechniques–Interaction Techniques

1. Introduction
Characters, such as those in animated films or computer
games, or even autonomous robots interacting in the realworld, are becoming increasingly common in everyday life.
Having convincing, believable personalities and behaviors isvery important for these characters, as it strengthens communication and suspension of disbelief, and ultimately results in a more rewarding, engaging, and comfortable experience [Bat94, Bre02, RN96]. In particular, it is critically important that interactive characters react convincingly to realtime user input while maintaining a coherent personality. For
example, an aggressive merchant in a video game may chaseafter the user’s character, a shy cleaning robot may hide fromhumans while cleaning, and a pet robot dog may jump happily when its owner returns home.
Programming a character’s real-time interactive behavior
is a difficult problem, particularly when trying to achieve acertain personality or interaction style. The logical, step-bystep state-machine style endorsed by conventional programc The Eurographics Association 2008.

Figure 1: First the behavior of a reactor is demonstrated in
response to a main character’s behavior. At run time, the reacting character’s behavior is synthesized, reproducing thepersonality and emotion demonstrated in the training.
ming languages is good for specifying goal-oriented actions,but is not directly well-suited to the design of human-liketraits such as personality and emotion; consider how difficult
it is to program interactive shyness or anger using a series oflogical states, rules and steps. Currently, programmers haveto implement complex models to achieve believable results.

J. E. Young & T. Igarashi & E. Sharlin / Puppet Master

Artists, however, such as actors, puppeteers, and so forth,have an incredible ability to develop interactive charactersand personalities in various performance-based mediums
such as puppet shows, plays, computer avatars and evenremote-controlled robots. Unfortunately, conventional tools
available to create autonomous interactive behaviors are often inaccessible to non-technical artists, in part because of
their algorithmic nature and their incapability to explicitly
express character personality and emotion. The result is thatinteractive behaviors are generated primarily by computer
software engineers using logic-based algorithms, with results often being predictable and boring.
In this paper, we introduce a programming-bydemonstration approach to make the design of interactive
character behavior accessible to artists. In the training phasea user (or two working collaboratively) demonstrates paired
motion of two characters, with one character reacting to theactions of the other. At run-time, the end-user controls themain character and the system generates, in real time, the reactive behavior of the other character with the characteristics
observed in the training data (Figure 1). Demonstrating aninteractive behavior allows artists to encapsulate personality
and emotion that they may not be able to logically explainusing computer algorithms. Furthermore, training is quick(average 33s in our study) and generation is done in realtime without preprocessing.
Our behavior-synthesis algorithm is an extension of theimage analogies algorithm [HJO∗ 01], which learns static
image filters from example image pairs and applies them toa new input image. Similarly, our system learns reactive behavior from an example pair of motions and applies it to a
new input motion. Our paper describes how we extend theoriginal method to work for real-time, dynamic, reactionary
locomotion behavior. Specifically, we introduce meaningful
behavior-related features, a method for balancing between
the similarity and coherence metrics, and we also separately
synthesize general motion trajectory and motion texture, integrating them during the final stages of motion synthesis.
We built two prototype systems, one using a standard
mouse as input and the other using a tabletop system withphysical pucks tracked by a motion capture system. The
tabletop system allows for simultaneous control of both themain and reacting characters, and allows the user to control
characters’ orientations. We ran a user study using the tabletop system asking one group of participants to design a setof character behaviors and the other group to interact witha set of designed behaviors. The results show that novice
users can design interactive behaviors quickly using our system and successfully convey personality and emotion in theform of interactive behavior.
2. Related Work
There has been a great deal of work that aims for lifelike, convincing interactive behavior. A common approachis to explicitly program the behavior model [BG95, Mae95,
Rey87], where the programmer explicitly defines what to do
for particular input scenarios. These systems require an understanding of the underlying algorithm and so are less accessible to the general artist, and do not support direct andintuitive design of emotion and personality.
Programming by demonstration was originally used
to automate GUI operations [Cyp, MWK89, PG96], e.g.,
Pavlov [Wol97], explicitly for interactive agents, defines
the low-level stimulus-response behavior of the agent using
logical event sequences and conditionals. Similar work byDinnerstein et. al [DE05, DEV] focuses on collisions, planning and goals, and requires the presence of a programmer.
While successful, these systems do not provide tools to represent personality and emotion in the same way we do. Several systems design animation by performance demonstration [DYP03, HOCS02, IMH05b, IMH05a, TBvdP04] or apply the idea to robotic motion [FSMI00,RPI04]. These, however, focus on the playback of the demonstration and do notrespond intelligently to user interaction.
Other systems focus on synthesizing new motion froman often large, pre-processed example database in realtime [LL04, LCL07, WH97]. While some systems interactively respond to user input (joystick control, moving obstacles, other characters, etc), the mapping from the user input
to the output is explicitly (and often tediously) defined by
the programmer. Furthermore, the target of these systems(e.g., [HGP04]) is primarily the physical accuracy (punch,jump, walk, collision avoidance, etc.), not the explicit designof personality emerging from interactive motion.
Research in human-robot interaction is shifting fromviewing robots as tools to designing affective robotic interfaces [Bre02,Nor04]. Robots are active participants in users’
physical environments, and so the design of their form, posture and movement play a dramatic role in the quality of theresulting interaction [MMMI05]. Designing robotic actions
and movement by demonstration has been investigated extensively (for example, [Bre02, FSMI00, RPI04, MMMI05])
with efforts ranging from simple movement playbacks tocomplex integrated actions. However, these methods target
goal-oriented design of robotic pose and motion, and do not
explicitly handle emotion or personality. Finally, it has beenshown that observers attribute emotion to simple motions
and actions [ABC, Kas82]. In our work, we enable designers to control (through demonstration) what sorts of motions
and actions will be perceived.

3. System Overview
The ultimate goal is to allow the intuitive design of all aspects of behavior (gestures, facial expressions, eye movement, pose, etc) to create believable whole-body characters.
As initial exploration this paper focuses on character locomotion (movement path). We implemented two interfacesc The Eurographics Association 2008.

J. E. Young & T. Igarashi & E. Sharlin / Puppet Masterthat enable designers to demonstrate behaviors: a mousebased GUI and a tabletop Tangible User Interface (TUI).

forces an artist to map their intentions through the arguably
less-than-intuitive mouse interface, separating the artist action and perception space [FBR86, SWK∗ 04].

3.1. Mouse-Based GUI

Our Tangible User Interfaces (TUIs)-based tabletop design (see Figure 3) allows artists to use physical objects (theTUIs) as handles to demonstrate behavior. The TUIs offer
immediate physical feedback of the system state, providing intuitive interaction not possible with traditional interfaces [IU97]. Using TUIs also allows for character orientation, and multiple TUIs can be used for simultaneous training of both the main and reacting character behaviors, either
two-handed by one designer or by a pair of collaborators.

The mouse-based GUI (Figure 2) works on any regular PC.
The standard mouse, however, lacks a rotation sensor and isdesigned for use by a single-user. This means that the main
character and reactor behaviors must be trained sequentiallyand that the direction a character is looking cannot be explicitly specified, e.g., a character cannot move sideways orback away (here, look direction is matched to movement direction). With sequential training, the designer first demonstrates an example input path to represent the main character. Then, the system replays the main-character’s motion
path while the designer demonstrates the sample reaction.
For behavior generation, the user simply controls the main
character and reaction is generated in real time.
3.2. Tabletop Tangible User Interfaces (TUIs)
Demonstrating interactive behaviors is reminiscent of actingor puppetry; actions generally done in the physical world,away from traditional computers. Using a mouse mapped
to an on-screen avatar removes this direct physicality and4. Algorithm
During training, the system simply stores the paired motion
data. During generation, the algorithm compares the runtime situation between the main and reacting character tothe training data. The training most similar to the current situation is used to direct the generation of the next character
output; the overall generation is based on a mix of training
data from various source locations. This method reacts immediately to changing user input and avoids large chunks ofconsecutive, static replay of training data while maintaining
the style and properties of the demonstration.
The key problems to be solved are to a) select a similarity
metric that matches behavioral similarity as end-users may
recognize it, and b) ensure the generation algorithm maintains the characteristics and textures given in the training
data. All of this must happen in real time.
4.1. Algorithm Formalization

Figure 2: The mouse GUI. Notice the large work area andlack of parameters and settings.

The variables I, R represent time-series data for the main (I)and reacting (R) characters. Following, I t , Rt are the training time-series data used by the generation system, recorded
from the demonstration. Pseudo code is given below: BestMatch() finds training data most similar to the current situation, and Generate() uses the selected data to direct thenext output action. New user input is not used until the subsequent iteration because features used (e.g., relative character position) require data for both characters. Our implementation operates at 40Hz so this delay is negligible.
Loope = BestMatch(I t , Rt , I, R)
newMovement = Generate(e)R.append(newMovement)
I.append(getNewInput())
4.2. Data and Features

Figure 3: A user interacting with the Vicon TUI system.

c The Eurographics Association 2008.

Our data is a time-dependent array containing the location,
x,y, and the direction d of each entity at each time point.
We extract various features that decide what characteristics

J. E. Young & T. Igarashi & E. Sharlin / Puppet Masterof behavior will be matched. We generally ignore worldcoordinates and focus on the relationship between the twoentities and changes in local state, and these features areevaluated over a time window to encapsulate a trend overtime. We explored many features not discussed here (e.g.,
direct path data, distance and delta distance between characters), and settled on the following features, as illustrated in
Figure 4. We omit a detailed discussion on the omissions forbrevity. The use of these features is different for each algorithm step as outlined in each respective section.
Velocity – the magnitude of the vector between an entity’s
position and its previous one. This captures speed andacceleration-related aspects of behavior such as different
reactions for stopped, accelerating, or slow input.
Relative position – position of the reactor in relation to themain character’s position and look direction (coordinate
space). This captures relational behavior such as following, circling, and approaching. One scalar value per axis:
representing how much the reacting character is behind or
in front of, and to the left or right of the main character.
Normalized look direction – look direction normalized to
movement direction, with 0 pointing forward. This is therelationship between where a character is looking andmoving, e.g., if it is backing up or moving sideways.
Relative look direction – the difference between the entities’ look directions. This captures turning away shyly
when observed or aggressively facing an opponent.
Absolute Movement direction – the vector from the previous world-coordinate position to the current one. This
does not involve the other entity but is used in generation(Section 4.4) to add high frequency texture to the output.
∆Direction – change in direction from one step to the next,
represents the shape of the locomotion path (not in relation to the other entity). This feature helps to identify similar movement shapes and styles such as shaky or smooth.

erated motion sequences. This metric has two key components: overall situational similarity and generated path coherency. These are checked in parallel and combined in eachstep over a given movement-history neighborhood n (40samples (1s) in our implementation).
Situation Similarity – This is based on the relationshipbetween the two entities, using the relative position (inputcharacter centric), relative look direction, and velocity features. It compares the n most recent pieces of user input andgenerated output from I, R to a moving window of size nover the training data I t , Rt (Figure 5(a)). At each window
location the features from each of the four paths form multidimensional vectors and corresponding vectors (I vs I t , R vsRt ) are compared using Euclidean distance squared. These
distances are then summed over the window, providing ameasure of similarity at that window location. A smaller
value represents a better match.
Generated Path Coherency – This emphasizes theshape, style and features of the generated path R in relationship to the trained Rt while putting less emphasis on the relationship between entities I and R. This relationship is still
important, however, as some aspects of coherency dependon the relationship between the entities, such as when the reacting entity wants to finish a circle around the main entity
it must properly follow as the main character moves. This
metric uses normalized look direction, delta direction, relative position, and velocity. When there is no training datathat matches well to the current inter-entity situation (i.e.,situation similarity is weak) generated path coherency helps
to ensure a generation that matches the characteristics of Rt .
This metric compares the recently generated data from R
over n to the regions in Rt that were used to generate the recent R (the entire Rt is not searched). That is, given recently
generated elements Rk over the n, the source neighborhoodin Rt that was originally used to generate Rk is compared tothe most recently generated R (Figure 5(b)). The intuition

4.3. BestMatch (Similarity Metric)
Our similarity metric is heavily based on Image Analogies [HJO∗ 01, HOCS02] but applied to dynamically gen-(a) Situation Similarity compares recent real-time data to theentire dataset.

(b) Generated Path Coherency examines the regions of the
training data recently used in generation.

Figure 4: Data features: All features except relative position
are on both entities, but only shown on one for image clarity.

Figure 5: Metrics derived from Image Analogies [HJO∗ 01]

c The Eurographics Association 2008.

J. E. Young & T. Igarashi & E. Sharlin / Puppet Master

here is to continue a patch from a previous similarity match
if the current similarity match is weak.
Similarity Balancing – The Image Analogies algorithm [HJO∗ 01] combines the two similarity metrics by statically weighting them with a coefficient k to add bias; themetric with the best weighted score is selected for that step.
This did not work with our application and resulted in a
problem we call coherence loops: when coherence match isused to generate output for several consecutive steps thenthe result of generation, by design, will be increasingly similar to the training data. The improving coherence match is
eventually exclusively used, with situation similarity being
ignored, and the reacting entity starts to loop through sections of Rt . This issue does not occur in Image Analogies
and Curve Analogies because all data is given at the beginning, allowing the use of multi-resolution approaches. Multiresolution is difficult in our system, however, as we are generating in real time and cannot look ahead in our input data.
We initially tried to mesh both metrics in several ways (e.g.,
average, trend-vs-detail, staggered sample rate), but did not
get satisfactory results. Our solution is given below, but thisis a rich area for future work.
We change the previously-static weight coefficient k to a
dynamic value to target a situation-similarity-to-coherency
match ratio t. Then, k is automatically tuned each generation step to bias the results so that over time we keep tbalanced. In our implementation, we use a 1:1 target ratio and k is tuned linearly. A similar algorithm is used in
texture synthesis systems to match the overall color histogram [KFCO∗ 07]. Following, the data from Rt immediately following the source region is passed to the generation system. One problem with this balancing approach isnoise. Instability in the similarity metrics (jumping between
regions) and switching rapidly between situation similarity
and coherence can cause large, rapid variations in the source
data passed to the generation system, resulting in distracting
rapid character movements. We explain our solution below.

4.4. Output Generation
The generation system receives the piece of training data(target data) to be used from BestMatch and generates thenext entity output. The naïve approach is to simply copy thisdata directly to the output as in texture synthesis. The problem with this is that many features depend on history andare relative to the other entity and it is impossible to solve a
movement that matches all features. Also, when the training
data jumps between drastically different states over consecutive steps this approach does not provide a meshing mechanism to generate intermediate data. These jumps suggest thattransitions are missing from the training data, and the generator function must handle discontinuities while maintaining
the texture, personality, and character that was demonstrated.
Our generation approach, a key technical contribution ofc The Eurographics Association 2008.

this paper, is to decompose emotion (the motion) into its
low-frequency (intentional move to certain relational position) part and high-frequency (texture of the motion) partand treat them separately. While Fourier analysis has been
done on motion paths before (e. g., [UAT95]), we are the firstto use it in terms of emotion and personality.
General Trajectory Generation – Motion is generatedusing relative position, normalized look direction, and velocity. Normalized look direction is copied directly to output, and a vector is constructed to move the entity from its
current location to the target relative position and then scaled
to the target velocity (Figure 6). Although the entity moves
toward the target position rather than be at that position, thevelocity scaling in combination with the high generation rate
helps to create very convincing results. Here we deal with thenoise resulting from the BestMatch instability by applying a simple linear smooth (average) over a history of threesamples. The results of this are very convincing and resultin a more stable, consistent generation. The problem, however, is that by removing the high-frequency noise we also
remove the high-frequency data, such as the movement detail and texture. We implemented a fix for this described in
the next section.
Another problem is that, even with smoothing, normalized look direction is very noisy. This happens because of thenature of the normalized look direction itself: if a character
keeps a static look direction in the world, but rapidly changes
movement direction, the normalized look direction (basedon movement direction) changes rapidly between drasticallydifferent values. An entity moving rapidly forward and thenbackward has data that alternates between 0 and π. Our solution to this is to limit rate of change of the actual worldcoordinate look direction. This lowers the amount of noisein resulting look direction, but some jitter remains. This is
an important problem for future work.
Detail Incorporation – To restore detail removed bysmoothing we do frequency analysis using Haar wavelets,
extracting the high-frequency detail from the target and directly incorporating it into the output. We apply Haar decomposition on the motion direction feature as this captures
path texture irrespective of velocity. A single application of
the discrete Haar decomposition scales our path data to half
resolution and stores the removed high-frequency detail separately. This gives a frequency cut at fs /2 where fs is the

Figure 6: To avoid drastic jumps, the reactor moves towards
the target position with the velocity taken from training data.

J. E. Young & T. Igarashi & E. Sharlin / Puppet Mastersampling rate: given k cumulative decompositions, this cut
is at fs /2k . The resulting k high-frequency datasets (one per
decomposition) can be re-composed to form a single highfrequency-only signal that we use in our generation. Our system uses four-level Haar decompositions, a frequency cut of
fs /16, or about 2.5 samples /s We found this to capture sufficient detail without affecting general trajectory.
The high-frequency data from the target is used to perturb
the generated (but smoothed) trajectory. While smoothingcompensated for BestMatch instability in trajectory generation, high-frequency source data cannot be smoothed in
the same fashion and so this instability remains, where interweaving detail from rapidly alternating training locations
results in noisy output not coherent to the training. Our solution applies the detail in patches of 16 samples: a patch isused in subsequent steps until the end when a new patch is
selected. These are 0.4 seconds long so the delay betweenchanged behavior and matching path detail is minimal.
5. Evaluating Puppet MasterOur evaluation consisted of two parts. The first part, theartist study, asked participants to design new behaviors using our system. The second part of the study, the end-user
study, asked participants to interact with behaviors createdin the first study. Our goals were to identify weaknesses in
our algorithm and interface, to get general user feedback andto determine how much (and what sorts) of characteristics,
emotions, and personality traits are captured by our system.
We initially conducted a pilot study to evaluate the study
protocol and procedure. 5 participants (2 female, 3 male)joined the artist pilot and 2 participants (1 male, 1 female)joined the end-user pilot. These pilots exposed language andquestionnaire wording that was confusing or strongly biased
users toward particular responses.
5.1. Experimental Testbed
The experiments used the tabletop TUI interface (Section 3.2) running a Pentium 4, 3.0 GHz PC, maintaining
40fps for up to 80s of training data (a behavior generally
requires less, as described below). We used the graphics inFigure 7 as static textures and a SMART Technologies 4’10”
x 3’7” high-resolution (2800x2100) rear-projected tabletop
display with pucks on the top to control input (Figure 3). The
pucks are tracked at 100fps by a six-camera Vicon motiontracking system, and the character graphics are drawn on thetabletop surface directly below the pucks in real time.

with computers. In the artist study (2 female, 8 male),four participants reported artistic experience with three having formal training and one identifying herself as an artist,and three users reported basic animation experience. Ages
ranged from 19 to 32 (M=22.8, SD=3.8). In the end-user
study (4 female, 6 male), nine participants reported artistic
experience with five identifying themselves as artists, andfour users reported animation experience (two extensive).
Ages ranged from 19 to 27 (M=23.7, SD=2.71). All participants had no prior exposure to the system and no participantsfrom the artist study took part in the end-user study.
Procedure – The artist study explored how general users
can use our system to create interactive behaviors. In onehour sessions, participants were first asked to design five
particular interactive character behaviors given the following keywords: lover, bully, playful friend, stalker, and afraid.
Participants completed a short written survey about the resultand experience after each behavior. Following, we evaluatedthe internal validity of the design by loading the five created behaviors each participant created in a scrambled order(fixed across participants) and asking them to interact with,and recognize, each behavior. Participants were not notified
ahead of time that they would be revisiting their own designed behaviors.
The end-user study observed how general users react tothe behaviors created using our system, and whether a senseof character emotion and personality emerged. We subjectively selected five behaviors created by participants in theartist study (one per each of the five behavior types), and participants were asked to “interact with and explore the characters” for each behavior in a fixed order, and to “describe
the character” in a questionnaire. Care was given to avoiding
affective or anthropomorphic language when presenting thetask to the end users, avoiding words such as “personality”,“behavior”, and “emotion”. In the second part participants
were asked to interact with a set of “other” behaviors whichwere in-fact a scrambled set of the previous behaviors. This
time users were asked to match each of the behaviors to thelist of “correct” behaviors as given in the artist study.
5.3. Artist-Study Results
Eight of 10 users in this study identified 100% of their own
behaviors. Further, in 74% of the cases (using 5 point Likert)
users agreed or somewhat agreed they were satisfied withthe resulting behavior, and in 22% of the cases they neither

5.2. Methodology
Participants – Twenty students (10 per study) from varying disciplines were selected from our university populationand paid $15 for participation. All users reported some toextensive programming experience and strong confidence

Figure 7: The graphics used in our evaluation, designed
generically to avoid particular emotion or personality.

c The Eurographics Association 2008.

J. E. Young & T. Igarashi & E. Sharlin / Puppet Masteragreed nor disagreed. The mean training time of acceptedbehaviors was 32.5s (SD=18.0s, min=9s, max=85s). The average number of trials required before accepting a behavior
was 1.7 (SD=0.9, mode=1 at freq.=56%, max=4 trials). The
average amount of time a user spent testing a result before
accepting it was 70.0s (SD=68.2s). In 46% of the cases users
disagreed that the generated behavior felt mechanical with
26% neither agreeing nor disagreeing. In 48% of the cases
users agreed that the behavior felt human-controlled (42%somewhat) with 26% neither agreeing nor disagreeing.
In the post-test questionnaire, on seven-point Likert, all 10
artist users agreed (5 strongly) that they enjoyed using thesystem, while 7 disagreed that the system was frustrating touse (1 strongly, 2 somewhat), all reported that the characters
were fun to play with (6 strongly, 2 somewhat) and 6 users
reported that movement jitter was distracting. The two users
who failed to recognize their own designed behaviors werealso the only two users who did not use puck orientation
during behavior training, resulting in poor quality behaviors.
Four artist users were notably immersed in the interface.
Some made exaggerated faces, noises, and spoke to thecharacters while training. One artist used the “jaws” theme
while training the “afraid” behavior, and another commented“what a jerk!” when observing their “bully” character. Users
generally expressed excitement about and satisfaction withthe capabilities of the system: “the system responded accurately and behavior was smooth, human-like, with a human touch”, “it’s even a better stalker than I am!”, “it almost
looks as if someone is controlling it.”, “it did exactly as I
wanted! Very entertaining! (maybe it’s just me?)”, “nailed
it!”, “I like it! I can see its bright future”, “the playful friend
is a hoot!” Several users commented on the robustness ofthe system, and one user was excited that it “even reacted
consistently with what [he] thought of after the fact.” Also,
users enjoyed the tabletop system, finding it “super easy andintuitive to operate. Instant results.”
On the other hand, several users reported issues with thesystem, commenting on the resulting generation as well as
the simplicity of our system: “it felt a bit mechanical withsome movements”, “as complexity of behavior rises it feels
more mechanical”, “if you pause to catch your breath, thesystem takes it as deliberate behavior”, “I need to try more
complicated behaviors”, “this setup cannot interpret smaller
actions that well”, “he doesn’t have hands so I can’t punch”,“difficult to imagine what one pretty slime does to bully another pretty slime.” Further, six of the ten users had issues
with occluding the Vicon markers on the controller puck.

“not mechanical at all” (6) the average response across all
behaviors was 4.04 (SD=1.19,Mode=4 at 36% frequency).
On another scale ranging from “a human is controlling it”(1) to “somewhat lifelike” (3, 4) to “not lifelike at all” (6),the average response was 3.4 with a mode of 5 at 24%.
To our pleasant surprise, out of the 50 cases (5 behaviors across 10 participants), characters were identified using the exact keywords used in the artist study 9 times,
and another 10 times using very similar words (for example, “girlfriend” instead of “lover”, “naughty, trying to bugme” instead of “bully”). Out of the 10 participants, 2 did
not match any behaviors, 2 matched 1 behavior, 3 matched 2
behaviors, 1 matched 3 behaviors, and 2 users matched 4 behaviors correctly. Furthermore, in the open-ended questionnaires 52% of all end-users behavior descriptions were usingsocial and behavioral descriptions (28% purely social), 34%of all the descriptions were using mechanical language (18%purely mechanical), with 14% being roughly a half-half mix.
For the second part of the end-user study, participants wereasked to match the five behaviors against the original keywords used. The results are given in Table 1, with the diagonal showing the number of end-users, out of 10, whomatched the pre-designed behavior to its exact keyword.
On the final questionnaires, 4 users agreed that the characters were sometimes confusing (1 somewhat), 1 neither
agreed nor disagreed, and 5 users disagreed (1 strongly, 1
somewhat). One strong observation throughout the study
was that users tended to see social characteristics and usedanthropomorphic language. For example, end users mentioned: “the guy who kept sucker-punching”, “each one
could bring to mind some real-life analogy”, “he needs more
confidence”, “I liked the part when it came close to my character ... kind of like a dog who is happy to see you”, “He
keeps trying to either hit you or kiss you”, “like an annoying kid brother in my face”, “he [the stalker] seemed likehe wanted to approach me, but he was too shy”, “facing it
and watching it panic like it had been discovered somewhere
where it shouldn’t be was fun”, “she [playful friend] is likea little sister who wants to talk to me.“Users were asked on the final questionnaire to describe thethings they liked and disliked about each character. While

5.4. End-User-Study Results
In the first part of the end-user study users were simply asked
to interact and describe prototype characters (without being prodded to look at behaviors or emotions). Here, on a
six point scale titled “the character felt...” ranging from “extremely mechanical” (1) to “somewhat mechanical” (3, 4) toc The Eurographics Association 2008.

Table 1: How behaviors were matched to original designs.

J. E. Young & T. Igarashi & E. Sharlin / Puppet Mastersome of these comments were analysis oriented, such as“actions were vague, subject to interpretation”, many of thecomments referred to the participant’s opinion of the character’s personality. For example, for the afraid character(which stayed away from the participant’s character) one
user wrote “I didn’t really like anything, didn’t even give mea chance to get to know him”, and others complained thatit “tries to invade my personal space. I like a nice personal
space bubble”, or “it doesn’t feel friendly!”
Similar to the artist study, some participants commented
that the characters felt a bit fake when the jitter was too noticeable and several participants complained that the personalities were too simple: “the personalities were very blunt,they were easy to see”, “I wish they could touch each other”.
All end-user participants enjoyed the experiment (6 strongly
agreeing). 7/10 users reported the pucks frustrating to use(all of these users commented on how easy it was to occlude
the Vicon markers), with the remaining 30% disagreeing
or strongly disagreeing. However, several users commented
that the table was “easy to use” and “intuitive”.

6. Discussion
These results help to support our claims about Puppet Master. The fact that 80% of the artist participants recognized
100% of their behaviors and were satisfied with the results
suggests that our algorithm successfully supports some levelof expression, and captures sufficient personality-relatedcharacteristics for recognition by the designer. That this was
accomplished without training at on average 32.5s shows,even for outliers (e.g., the 85s case), our algorithm allows
artists to create recognizable behaviors in significantly lesstime than it would take to program. Finally, that this was
accomplished in on-average 1.7 attempts by novices showsthat designers are able to satisfactorily and easily create behaviors, and that the real-time re-training and generation enabled the artists to flexibly explore design possibilities.
The end-user part of our study demonstrated that in 38%
of the cases behaviors not only emerged but closely matched
the artist keywords, based on motion only (Table 1). We believe that this supports our claim that our algorithm captures
the personality and style of the demonstrated behavior. Further, the results in Table 1 seem to hint at crosstalk between
similar behaviors: for example, afraid and stalker are often mistook for each other while lover, bully, and friend are
rarely mistaken for stalker or afraid. This shows that, even in
the cases where behaviors are not matched properly, there is
still a strong component of feeling and style captured fromthe demonstrated data.
Both studies suggest a strong sense of user engagement.
The explicitly positive study results, the verbal excitement,
as well as the extensive use of social and anthropomorphic
language suggests that the participants were interested andmentally involved with the design process.

7. Limitations and Future Work
Our current implementation does not handle dynamics overa large time scale and will fail to accurately represent an angry character gradually calming down. Perhaps this could
be explored through high-level features and multi-resolution
searches. It is also useful to consider how this work can combine with other behavior models and systems for a multilevel solution, and to understand the absolute limitation of
our approach, that is, when high-level behavior changes may
be better designed using scripting and explicit states.
Accounting for environmental issues (wall, tree, etc)
would improve the versatility of Puppet Master. Related, extending to several simultaneous entities would allow swarmlike behavior with individual personalities, e.g., to train agroup of archers and knights to storm a castle. Systems thatlearns crowd behavior from examples [LCHL07, LCL07]
mainly focuses on collision avoidance while we want to allow artists to interactively design more intentional crowds.
Perhaps our work could be applied to the design of robot
behaviors, e.g, replacing physical pucks and tabletop displays with mobile robots such as curlybot [FSMI00]. Physical movement can give stronger impression of personality
and emotion than virtual characters, but introduces severe
constraints on motion (e.g. robot cannot jump to distant position). However, many of the techniques developed in thiswork such as similarity-coherence balancing and trajectorytexture separation should be applicable to real robots, too.
Our ultimate goal is to design all aspects of character behavior, not just locomotion, responding to various input suchas dancing to music, sword-swing against an opponent, andmeow noise of a cat responding other meow noise.
8. Conclusion
Believable, convincing, and stylistic interactive behavior isan important aspect of any computerized entity that must interact with humans, such as avatars, video game characters,
or even robots. Traditionally, the creation of such a system
has been left to logical, step-by-step computer algorithms;
tools generally out-of-reach for non-technical artists and illdesigned for the creation of stylistic behaviors. In this paper we presented the first system that enables the programming of interactive behaviors by demonstration with realtime generation, making the creation of believable, stylistic
interactive characters accessible to the non-technical artist.
Acknowledgements
This work supported in part by NSERC, iCore, JSPS, JST,and the University of Calgary.
References[ABC] A MAYA K., B RUDERLIN A., C ALVERT T.: Emotion from motion. In Proc GI ’96, pp. 222–229.
c The Eurographics Association 2008.

J. E. Young & T. Igarashi & E. Sharlin / Puppet Master[Bat94] BATES J.: The role of emotion in believable
agents. Comm. ACM 37, 7 (July 1994), 122–125.

crowd simulation. In SCA ’07 (Switzerland, 2007), EG,
pp. 109–118.

[BG95] B LUMBERG B. M., G ALYEAN T. A.: Multi-level
direction of autonomous creatures for real-time virtual environments. In Proc. SIGGRAPH ’95 (1995), pp. 47–54.

[LCL07] L ERNER A., C HRYSANTHOU Y., L ISCHINSKI
D.: Crowds by example. Comp. Graphics Forum 26, 3(2007), 655–664.

[Bre02] B REAZEAL C.: Designing Sociable Robots. MIT
Press, 2002.
[Cyp] C YPHER A.: Eager: programming repetitive tasks
by example. In Proc. CHI ’91 (NY), pp. 33–39.

[LL04] L EE J., L EE K. H.: Precomputing avatar behavior
from human motion data. In Proc. SCA ’04 (Aire-la-Ville,
Switzerland, Switzerland, 2004), Eurographics Association, pp. 79–87.

[DE05] D INERSTEIN J., E GBERT P. K.: Fast multi-level
adaptation for interactive autonomous characters. ACM
Transactions on Graphics 24, 2 (2005), 262–288.

[Mae95] M AES P.: Artificial life meets entertainment:
lifelike autonomous agents. Comm. ACM 38, 11 (1995),
108–114.

[DEV] D INERSTEIN J., E GBERT P. K., V ENTURA D.:
Learning policies for embodied virtual agents throughdemonstration. In Proc. IJCAI ’07, pp. 1257–1262.

[MMMI05] M ATSUI D., M INATO T., M AC D ORMAN
K. F., I SHIGURO H.: Generating Natural Motion in an
Android by Mapping Human Motion. In Proc. IROS ’05(USA, 2005), IEEE, pp. 1089–1096.

[DYP03] D ONTCHEVA M., Y NGVE G., P OPOVI Ć Z.:
Layered acting for character animation. ACM Trans.
Graph. 22, 3 (2003), 409–416.
[FBR86] F JELD M., B ICHSEL M., R AUTERBERG M.:
Build-it: a brick-based tool for direct interaction, 1986.
[FSMI00] F REI P., S U V., M IKHAK B., I SHII H.: curlybot: designing a new class of computational toys. In Proc.
CHI ’00 (NY, USA, 2000), ACM, pp. 129–136.
[HGP04] H SU E., G ENTRY S., P OPOVI Ć J.: Examplebased control of human motion. In SCA ’04 (Switzerland,2004), EG, pp. 69–77.
∗[HJO 01] H ERTZMANN A., JACOBS C. E., O LIVER N.,
C URLESS B., S ALESIN D. H.: Image analogies. In Proc.
SIGGRAPH ’01 (2001), pp. 327–340.
[HOCS02] H ERTZMANN A., O LIVER N., C URLESS B.,
S EITZ S. M.: Curve analogies. In Proc. EGRW ’02(2002), pp. 233–246.
[IMH05a] I GARASHI T., M OSCOVICH T., H UGHES J.:
Spatial keyframing for performance-driven animation. In
Proc. SIGGRAPH ’05 (July 2005), pp. 107–116.
[IMH05b] I GARASHI T., M OSCOVICH T., H UGHES J. F.:
As-rigid-as-possible shape manipulation. ACM Trans.
Graph. 24, 3 (2005), 1134–1141.
[IU97] I SHII H., U LLMER B.: Tangible bits: towards
seamless interfaces between people, bits and atoms. In
Proc. CHI ’97 (NY, 1997), ACM, pp. 234–241.
[Kas82] K ASSIN K.: Heider and simmel revisited: causal
attribution and the animated film technique. Rev. Pers.
Soc. Pschol. 3 (1982), 145–169.
[KFCO∗ 07] KOPF J., F U C.-W., C OHEN -O R D.,
D EUSSEN O., L ISCHINSKI D., W ONG T.-T.: Solid texture synthesis from 2d exemplars. ACM Tran. Graph. 26,
3 (2007), 2:1–2:9.
[LCHL07] L EE K. H., C HOI M. G., H ONG Q., L EE J.:Group behavior from video: a data-driven approach toc The Eurographics Association 2008.

[MWK89] M AULSBY D. L., W ITTEN I. H., K ITTLITZ
K. A.: Metamouse: specifying graphical procedures by
example. In Proc. SIGGRAPH ’89 (NY, USA, 1989),
ACM, pp. 127–136.
[Nor04] N ORMAN D. A.:
Books, NY, 2004.

Emotional Design.

Basic[PG96] P ERLIN K., G OLDBERG A.: Improv: a system
for scripting interactive actors in virtual worlds. In SIGGRAPH ’96 (NY, 1996), ACM, pp. 205–216.
[Rey87] R EYNOLDS C. W.: Flocks, herds and schools: A
distributed behavioral model. In Proc. SIGGRAPH ’87(NY, 1987), ACM, pp. 25–34.
[RN96] R EEVES B., NASS C.:CSLI Publ., UK, 1996.

The Media Equation.

[RPI04] R AFFLE H. S., PARKES A. J., I SHII H.: Topobo:
a constructive assembly system with kinetic memory. In
Proc. CHI ’04 (NY, USA, 2004), ACM, pp. 647–654.
[SWK∗ 04] S HARLIN E., WATSON B., K ITAMURA Y.,
K ISHINO F., I TOH Y.: On tangible user interfaces, humans and spatiality. Personal Ubiquitous Comput. 8, 5
(2004), 338–346.
[TBvdP04] T HORNE M., B URKE D., VAN DE PANNE M.:
Motion doodles: an interface for sketching character motion. In Proc. SIGGRAPH ’04 (New York, NY, USA,
2004), ACM, pp. 424–431.
[UAT95] U NUMA M., A NJYO K., TAKEUCHI R.: Fourier
principles for emotion-based human figure animation. In
SIGGRAPH ’95 (1995), pp. 91–96.
[WH97] W ILEY D. J., H AHN J. K.: Interpolation synthesis of articulated figure motion. IEEE Comp. Graph. andApp. 17, 6 (/1997), 39–45.
[Wol97] W OLBER D.: Pavlov: an interface builder for designing animated interfaces. ACM TOCHI 4, 4 (1997),
347–386.