The Visual Computer manuscript No.
(will be inserted by the editor)

Generating Animation from Natural Language Texts andSemantic Analysis for Motion Search and Scheduling
Masaki Oshita

Received: date / Accepted: date

Abstract This paper presents an animation systemthat generates an animation from natural language texts
such as movie scripts or stories. It also proposes a framework for a motion database that stores numerous motion clips for various characters. We have developed semantic analysis methods to extract information for motion search and scheduling from script-like input texts.
Given an input text, the system searches for an appropriate motion clip in the database for each verb in the
input text. Temporal constraints between verbs are also
extracted from the input text and are used to schedulethe motion clips found. In addition, when necessary,certain automatic motions such as locomotion, taking
an instrument, changing posture, and cooperative motions are searched for in the database. An animationis then generated using an external motion synthesis
system. With our system, users can make use of existing motion clips. Moreover, because it takes natural
language text as input, even novice users can use oursystem.
Keywords computer animation · motion database ·natural language processing

1 Introduction
Recently, computer animation has been widely used inmovies, video games, TV programs, web graphics, etc.
Because computer animation is a very powerful tool
Masaki Oshita
Kyushu Institute of Technology
680-4 Kawazu, Iizuka, Fukuoka, 820-8502, Japan
Tel.: +81-948-29-7718
Fax: +81-948-29-7709
E-mail: oshita@ces.kyutech.ac.jp

to present a story, drama, or instruction, there are demands from non-professional people to create computer
animation. However, it is a difficult task because of two
main issues. The first issue is the difficulty of making
and reusing motion data. Currently, motion data aremainly created using motion capture or keyframe techniques. Either way, they are very time consuming andrequire professional skills. Although there are demands
for reusing existing motion data, this is difficult becauseof the lack of a system for storing and searching large
amounts of motion data. Because there can be various
motions of various characters, it is difficult to manage
them in a standard file system or database. Currently,
most motion data are created from scratch for individual scenes and are thrown away without reuse. The
second issue is the limitation of current animation systems. A computer animation can be created by combining a number of existing motion clips using animation
software such as MotionBuilder, Maya, 3ds Max, etc.
However, it is difficult for novice users to utilize such
software, because handling motion data is tricky andthese systems require training.
To address these issues, we developed an animation
system that generates an animation from natural language texts such as movie scripts or stories (Fig. 1). We
also developed a motion database that stores many motion clips for different characters. When an input text
is given, the system searches for an appropriate motion
clip from the database for each verb. Temporal constraints between verbs are also extracted from the input
text. The searched motion clips are scheduled based on
the temporal constraints. In addition, when necessary,
some automatic motions such as locomotion, taking an
instrument, changing posture, and cooperative motions
are searched from the database. The system outputs
a motion timetable which consists of motion clips and2(a) Input text
Neo waves to Jack. At the same time, Jack takes the red bottle. Jack hits Neo with it.
(b) Output motion timetableJack

locomo takeNeolocomowave

locomo(c) Generated animation
hit
staggertime

Fig. 1 Example of our system. (a) Input text. (b) Searched motion clips and their execution timings. (c) Generated animation.

their execution timings. An animation is then generated using an external motion synthesis system. Using
our system, even novice users can create animation bymaking use of existing motion clips.
There are many possible applications of our system. Recently, in movie production, simple animations
are created before production to check camerawork,
screenplay, necessary visual effects, etc. These animations are called “previsualization” or “animatics”. They
are also often created for the scenes in which no computer graphics are involved. Using our system, evendirectors or writers who are not professional animators can create an animation very quickly. Moreover,
our system can be used by non-professional people whowant to make an animation but do not have professional
skills. It can also be used for children to visualize a story
to make it interesting and easy to understand. Our system can be used for movie production. Even though
animators want to add more details to the output of
our system, our method is much easier than making
animations from scratch.
In this paper, we propose a motion frame that contains meta-information about a motion clip, an objectoriented database framework for storing a number ofmotions of a number of characters in a hierarchicalstructure, natural language analysis methods that are
specialized for extracting motion related descriptions
from an input text, and scheduling of multiple motions
based on the temporal constraints in an input text. In
addition, we have done preliminary experiments whichshowed that our system generates expected results fromvarious input texts.
This paper is an extended version of our previous
work [1]. As explained in Section 5, we have mainly
extended our natural language analysis methods to enable our system to handle various expressions in input
texts. Based on the experiments presented in Section 8,
87% of the verbs in a sample movie script can be dealt
with using our methods and represented as motions, al-though 78% of these were handled by the system before
the extension [1].
The rest of this paper is organized as follows. Section 2 reviews related work in the literature. Section 3
gives an overview of our system. Sections 4, 5, 6, and 7
describe our methods used in the framework of the motion database, natural language analysis, motion search
and motion scheduling, respectively. In Section 8 someexperimental results are presented together with a discussion thereof. Finally, Section 9 concludes the paper.

2 Related Work
Generating animation from natural language texts has
been a challenge. Many research groups have tackled
this problem. The SHRDLU system, which was developed by Winograd [2], is known as the pioneer. Using
SHRDLU, a user can give commands to a robot using
English in an interactive manner, and make it arrange
objects in a scene. However, the types of commands
were very limited.
Badler et al. [3][4] developed virtual agents that follow natural language interactions. They proposed Parameterized Action Representation (PAR), which has
a similar purpose to the motion frame in our research.
The PAR has more complex information such as precondition and achievement. The motion generator ofeach PAR is programmed using a state machine. It
can use motion data or any motion generation methods. However, specifying detailed information and constructing motion generators are very time consuming.
Tokunaga et al. [5] developed the K2 system, which
has similar goals to Badler et al. In their system, agentsare controlled via spoken language. Their research israther focused on solving the vagueness of natural language instructions. They use case frames [6] to search
for motions. Unlike our work, they use all cases that areused in linguistic analysis. The interpretation of eachcase is left to the user who adds the case frame han-3dler. The motion generator for each case frame must be
manually programmed by the user.
These previous works aim at developing intelligent
agents that understand natural language instructions
and make plans to execute them. However, the systems are very complex, and many rules are required.
On the other hand, our system aims to reuse existing
motion data easily and efficiently. The motion frame inour work contains just enough information to search forappropriate motions that match natural language texts
and it is easy to describe. We believe that our system
is more practical.
Lu and Zhan [7] developed an animation production
system that includes story understanding, plot planning, act planning, camera planning, etc. Although theirsystem takes simple Chinese as input, it requires a great
deal of additional knowledge, including not only caseframes but also many dictionaries, templates and rules.
Sumi et al. [8] developed a system for visualizing
short stories for children. The system extracts keywords
from an input text, and chooses an appropriate scene,characters, and motions from a database. It simply plays
a motion that matches the keywords. Although a user
can add motion data to the system, the system cannot
select motions appropriate for the objects or characters and cannot generate interactions between characters and the scene.
There is very little research that deals with motion
scheduling from natural language texts. The above systems simply execute motions as instructions are given
or events happen, and no scheduling is considered. However, in order to execute multiple motions of multiple
characters as instructed by an input text, the execution
timing of the motions must be coordinated. Baba et
al. [9] developed a system for generating an animationthat satisfies temporal and spatial constraints given by
natural language texts. The system determines appropriate initial positions of the agents and objects thatare specified in the input text. However, the motions of
the agents and motion scheduling were not considered.
Coyne and Sproat [10] developed WordsEye, whichconverts natural language texts to a scene. Because
their purpose is to generate a still image, when a character motion is indicated in a given text, the system
simply chooses a pose for the action from the database.
There have been various studies on generating acharacter’s gestures for a monologue or conversation[11]. These methods generate motions by composing
short fragments of motions based on signal processing of
the input speech rather than by interpreting the meaning of the speech.
There are also animation engines that support some
script language such as Improv [12] and Alice [13]. How-ever, it is still difficult to program the agents and to
make use of a large amount of existing motion data. In
addition, markup language formats for describing animation including scenes, characters and actions havebeen proposed [14][15] . However, they are difficult to
describe by hand. The animation files should be created by using specific authoring software. Moreover, it
is difficult to add and reuse motion data using such fileformats and authoring software.
There are many motion synthesis methods which
generate new motions from a small number of motions[17][18]. However, they require a manual setup for eachmotion module. It is difficult for end users to add new
motion modules. Although currently our system selects
one motion from the database, it is possible to extend
our system to blend a number of selected motions basedon quantitative motion query parameters such as contact position.

3 System Overview
In this section, we explain the overview of our system(Fig. 2) and data representation (Fig. 3).
When an input text is given to the system, natural language processes (syntax analysis and semantic
analysis) are applied first. The syntax analysis is theprocess of converting a plain text to a tree structurewith phrase tags and dependencies. Fig. 3(b) is an example of the analyzed tree which is computed from an
input text (Fig. 3(a)). The type of each phrase and thedependency between phrases are determined. For example, S, NP, VP and PR in Fig. 3(b) represent sentence,noun phrase, verb phrase and preposition, respectively.
The semantic analysis extracts information about
motions described in the input text from the tree structure. A query frame contains information for the motion search. One is generated for each verb in the text.
The temporal constraints contain information about execution timing between verbs. For example, QF1∼QF3and TC1∼TC2 in Fig. 3(c) represent query frames andtemporal constraints, respectively.
Based on the temporal constraints, motion scheduling determines the execution order of each motion clip,
which corresponds to each query frame as shown in Fig.
3(d). Note that exact execution times are not decided
at this point, because the duration of each motion is
not known until motion clips are searched from thedatabase and automatic motions are added later.
The motion search is applied for each query frame.
In addition, when it is necessary, automatic motions
are inserted before the motion. Finally, motion clips
and their execution timings are passed to the motion

4
Input Text

Syntax Analysis

Analyzed Tree

Semantic Analysis

Query Frames
Temporal Constraints

Motion Scheduling

Scheduing Information+ Query Frames
MotionDatabase

Motion Search

Scheduing Information+ Motion Clips
Scene Information
Characters and Objects

Motion Synthesis

Animation

Fig. 2 System overview.

(a) Example of an input text
Neo waves to Jack. At the same time, Jack takes the red bottle. Jack hits Neo with it.
(b) Analyzed tree
Neo waves to Jack. At the same time, Jack takes the red bottle. Jack hits Neo with it.
NP
NP
NP
NP
NP
NP
NP
NP
PP
PP
PP
VP
VP
VP
S
S
S
(c) Query frames and temporal constraints
QF1
agent: Neomotion: wave
target: Jack

QF2
agent: Jack
motion: taketarget: red bottle

TC1
Synchronized
QF1 = QF2

TC2
Serial
QF2 -> QF3

QF3
agent: Jack
motion: taketarget: Neo
instrument: red bottle

synthesis module as a motion timetable, as shown in
Fig. 3(e).
The motion synthesis generates an animation bysmoothly connecting given motion clips. The interactions between characters and between a character andobjects are handled by this module based on the information that the motion clips have.
The scene information contains characters and objects and their initial states, including postures, positions, and orientation. Each object has certain object
information including names, a default contact point,
alternative contact points and their names. For example, a desk object has “desk”, “table”, etc. as its names.
An object also has sets of pairs consisting of a partname and its position (e.g., “above”, “under”, “side”,etc.). This information is used to search for appropriate motions and determine appropriate contact positions according to an adjective that is used in the input text. In addition, an object has a default position
which is used when no adjective is specified. This kindof object information is commonly used in similar approaches [5][10]. In addition, a scene also has default
entering and leaving points as the default goal locations
for locomotive motions (see Section 5.5). Currently, oursystem assumes that the scene information is provided
in advance by the user.
The scope of this paper is the components in thedotted box in Fig. 2. There are many tools for syntax analysis that can be used with our system. The
Stanford parser [19] is used for our implementation.
For motion synthesis, our system uses an external animation system [20]. The system generates continuous
motions from given motion clips and their execution
timings. The system determines an appropriate synthesis method for each transition based on the constraints
between the foot and the ground during motions. Alternatively, another commercial animation system suchas MotionBuilder, Maya, 3ds Max, etc. can be used.

(d) Scheduling information and query frames
Jack

QF2

Neo

QF1

QF3
serial

4 Motion Databasesynchronized(e) Output motion timetable(Motion clips and their execution timings)
Jack

locomo takeNeolocomowave

locomohit
staggertime

In this section, we describe the representation of motion data. We first explain the case frame that is usedin natural language processing. Then, we explain our
motion frame, which is inspired by the case frame. We
also describe our database of characters and motions.

Fig. 3 Example of data representation.

4.1 Case Frame
The idea of a case frame was proposed by Fillmore [6].
A case frame represents the role of a verb. Each case of

5

a case frame is a phrase that represents an aspect of theverb. Typically a case frame has the following cases:– Agent: the person who performs the motion.
– Experiencer: the person who experiences something.
– Object: the object that an effect is caused to during
the motion.
– Instrument: the object that causes an effect during
the motion.
– Source: the source or origin of the motion.
– Goal: the goal or target of the motion.
– Time: the time when the motion is performed.
– Location: the location where the motion is performed.
Each case needs to be a specific type of entity. Somecases are mandatory for some verbs. A verb that has
different roles depending on context has multiple caseframes.
In general natural language processing systems, a
procedure to select a case frame for an input text is asfollows. First, based on the types and dependency ofphrases in the analyzed tree, candidate cases of eachphrase are determined. By searching for case framesthat match the candidate cases, the most appropriatecase frame and all its cases are determined.
The case frame is a good way to extract and represent the meanings of texts. The case frame is widelyused in many research papers such as [5][10]. However,the case frame is not suitable for representation of motion data for animation. From the view point of motion
representation, each case has different roles dependingon case frames. For example, the “object” case of a caseframe could be an object that the character uses or another character that the character’s motion causes aneffect on. Moreover, the case frame does not contain information about postures and contact positions, which
are important for selecting motions.

4.2 Motion Frame
We propose a motion frame, which contains the information about a motion clip. The motion frame is inspired by the case frame. However, we define the itemsof the motion frame based on importance when we
search for a motion according to input texts.
There are many kinds of verbs in general English.
However, our system handles only action verbs that involve a physical motion, in other words, verbs that canbe visualized as an animation. Other kinds of verbs
such as non-action verbs (e.g., “think”, “believe”) or
state verbs (e.g., “know”, “exist”) are ignored in oursystem, because they are difficult to represent by a motion clip. Action verbs are categorized into intransitive,

Item
Agent
Names of Motion
Instrument
TargetContact Position
Target Direction
Initial Posture
Adverbs

Value
human
take, pick up, get
NULL
appropriate size and weight ranges
hand position of contact
NULL
standing
slowlyFig. 4 Example motion frame of “taking-an-object”.

transitive, and ditransitive verbs. Intransitive verbs involve no other object (e.g., “he runs”). Transitive verbs
include one target object/character/position (e.g., “heopens the door”, “he hits her”, “he walks to the door”).
Ditransitive verbs include two target objects (e.g., “hegives her the book”, “he cuts the bread with a knife”).
For distractive verbs, one of the two target objects shouldbe the object that the character possesses. We call suchobjects “instruments”. Therefore, action verbs have at
most one “target” object /character/position and at
most one “instrument” object. We use them as items ofa motion frame instead of cases in a case frame. In addition, contact position is used to select a motion thatfits the environment and previous motions.
The items of the motion frame are as follows. An
example of a motion frame is shown in Fig. 4. Notethat some items may not have any value depending on
the motion.
– Agent Magent ref : The reference to the character inthe database who performs the motion.
– Names of motion Mmotion strings : The set of verbs
that represent the motion. When a verb in the input text matches one of the motion names, the motion frame will be a candidate for the verb. To handle ambiguity, a motion frame may have multiple
names. For example, a “taking-an-object” motion
may have “take” and “pick up” as its names.
– Instrument Minstrument ref , Minstrument params : The
object that the character uses in the motion. Thisis either a reference to an object in the database
Minstrument ref or the size and weight ranges of an
object Minstrument params . If the motion requires a
specific object such as “cutting with a knife”, the object should be specified as a reference to the instrument. Otherwise abstract conditions of an object
are specified. For example, if the motion is “poking
something with a long object”, then appropriate size
and weight ranges of the object are specified.
– Target: The reference to an object Mtarget ref or thesize and weight ranges Mtarget params are specified
in the same way as the instrument. If the target is a6character, the reference to the character is specified
in Mtarget ref .
– Contact position Mcontact vertical , Mcontact horizontal :the position of the end-effector when it makes contact with the target. A contact position is specified
when the motion involves contact with a target character or object. Vertical and horizontal positions arehandled differently. Because the horizontal position
can be adjusted by lateral movement (see Section
7.2), the vertical position is more important for motion selection. For example, if multiple “taking an
object” motions are in the database and an input
text “he takes the bottle on the ground” is given,
then based on the position of the bottle, the appropriate taking motion (e.g., “taking an object withsquatting”) will be selected. The contact position is
automatically computed from the contact information (see Section 4.3) of the motion data. The contact position is expressed in the local coordinates ofthe motion data.
– Target direction Mtarget direction : The direction of
the target. For some motion, even though the motion does not involve contact with the target, thetarget direction is important. For example, when“waving to a person” or “shooting a target” motion
is executed, the character should face the right direction. For some motion, both contact position andtarget direction are specified. For example, “sitting
down on a sofa” motion should make contact witha sofa from the front of the sofa.
– Initial posture Minitial posture f lag : the character’s
posture when the motion begins. Currently, it is represented as one of three states: standing, sitting, orlying down. The initial posture is used to select a
motion that matches the terminal posture of theprevious motion. In cases where no such motion isin the database, an automatic changing posture motion will be added (see Section 7.2 ).
– Adverbs Madverb strings : The set of adverbs represent the style of the motion such as “slowly” or“happily”.
Each item of motion frames must be specified bya user. However, this is not such a difficult task forusers. For each motion frame (each motion clip), theuser is asked to specify the agent, verbs, target, andinstrument. The agent is selected from the character
database. For the target and instrument, it is either
an appropriate object or agent that is selected fromthe database or the size and weight range of an object. When the motion involves a specific object (e.g.,“cutting with a sword”), the object should be selected.
Otherwise, object conditions are specified (e.g., “lifting
up a light object using one hand”). The contact posi-tion is automatically computed form the motion and its
contact information (see Section 4.3 ). The initial posture is also automatically computed from the motion
clip. As a result, specifying the items of a motion frame
is very easy.
4.3 Motion Data
Our system supposes that each motion is short and simple. A complex motion is difficult to represent by a motion frame. If a user wants to add a long motion to the
database, the motion should be divided into pieces.
Some motions involve an interaction with an object
or a character. This information is very important for
generating animation and for selecting motions. Therefore, it is specified on the motion frame. The contact
information consists of the contact type (hold, release,or hit), contact time (local time in the motion clip) andthe end-effector (e.g., right hand). This information is
also necessary for generating animation in the motion
synthesis module (see Section 6.2).
Some motions that interact with another character
cause the reaction of the other character (e.g., “Jackhits Neo. Neo falls”). Usually such cooperative motions
are captured or created at the same time but are stored
as separate motion clips. In our system, such cooperative motions are specified on the motion frame. If
a motion has cooperative motions and no cooperative
motion is indicated in the input text, the system automatically executes a cooperative motion (see Section
7.2). In addition, when two cooperative motions include
physical contact, the timings and the initial positionsof these motions are coordinated (see Section 7.1).
4.4 Character and Motion Database
We use an object-oriented framework for the characterand motion database. As shown in Fig. 5, each character is considered to be an object that has various
motions as its methods. A character inherits from abase character. A motion of the base character can beoverridden by another motion. The motions that are
not overridden are used as the motions for the derived
character. In this way, the hierarchy of characters andtheir motions are efficiently managed. A character caninherit from multiple base characters. All motions thatthe base characters have are used for the derived character. Since the motion that most closely matches aninput sentence is selected from the available motions,even if there are multiple motions with the same name,there is no problem with conflicts caused by the multiple inheritance.

7

between verbs. In the followings of this subsections, we
explain how the semantic analysis works.

Human+ Walk+ Take
+ Hit
…

Male

Female+ Take (specialized)
…

…

Jack

Trained Person+ Take (specialized)+ Hit (specialized)…

+ Walk (specialized)+ Take (specialized)+ Hit (specialized)…

Neo+ Hit (specialized)…

Fig. 5 Example of a hierarchical database of characters.

If a user wishes to create a new character, s/he simply adds the new character that inherits from a basecharacter or multiple base characters to the databaseand adds character-specific motions to that character.
Even if there are not many new motions for the new
character, the motions of the base characters are used.
In this way, users can add new characters very easily.
The database can be implemented in various ways.
If the characters and motions are implemented using
an object-oriented programming language (e.g., C++or Java), we would represent motions as objects rather
than methods and implement a mechanism of motion
inheritance on the character class, because it is practically difficult to handle motions as methods using suchprogramming languages.
5 Natural Language Analysis
Although natural language processing techniques haveadvanced in recent years, it is still a challenge to understand general texts, because it requires not only language processing but also a large knowledge of the world.
However, our system is supposed to take script-like text
and only motion-related descriptions in the text matter.
This makes the natural language analysis much easier than general natural language processing systems
such as machine translation or summarization systems.
Moreover, because scene information, such as characters and objects, is given in advance, we do not need
the same large dictionary required by general natural
language processing systems.
As explained in Section 3, the semantic analysis
takes an analyzed tree and generates query frames andtemporal constraints. A query frame contains information of a verb for the motion search. The temporal constraints contain information about the execution timing

5.1 Query FrameTo select a motion that matches an input text, we use
a query frame, which has the same items as the motion
frame, and whose items are determined by analyzing
the syntax tree of the input text (see Fig. 3(b)). Scene
information is also used to determine some items.
As explained in Section 4.2 , unlike generic semantic
analysis, motion searches only need a target and an
instrument for each verb. Therefore, we determine theseby applying the following rules to each verb in the inputtext.
– A verb is used as the name of motion of the query
frame Qmotion strings . If the verb is followed by apreposition or noun, then all sets of the verb andthe following word are also set to Qmotion strings ,because this could represent an idiom. Therefore,
Qmotion strings can contain multiple phases. For example, in “Jack falls back”, both the phrases “fall”
and “fall back” are set to Qmotion strings .
– If a noun represents a character in the scene andthe verb is dependent on the noun, the character isconsidered as the agent (subject) of the query frame
Qagent ref .
– If two nouns are dependent on the subject that theverb is related to, they are considered as the target Qtarget ref and the instrument Qinstrument ref .
(E.g., in “Jack gives Neo the book”, “Neo” is thetarget and “the book” is the instrument.)– If only one noun is dependent on the subject, it isconsidered as the target Qtarget ref .
– If a preposition phrase (e.g., “to Neo”) is dependent on the subject, it is considered as the target
Qtarget ref or the instrument Qinstrument ref depending on the preposition. If the preposition is“with” and the noun in the phrase represents anobject, the object is used as the instrument. Otherwise, the noun is used as the target.
– If the character is holding an object, the object isalso used as Qinstrument ref , even if it is not specified in the input text.
– If a phrase considered to be an adverb in the syntax
analysis is dependent on the subject, the phrase isused as one of the adverbs Qadverb strings that cancontain multiple phrases.
After the names of the target and instrument are determined, we obtain the reference or value of each itemfrom the scene information. We suppose that the characters or objects in input texts always exist in the scene.

8

Therefore, unlike general semantic analysis, by looking
up the scene information all nouns in input texts are
determined.
The target character or object that is indicated in
the input text is searched from the scene informationand the reference and position are set to the query
frame. When the target is a character and a body part isindicated in the text such as “She hit him in the head”,
the reference and position of the body part is set. Whenthe target is an object in the scene, the target size andweight are set in the query frame. The contact position Qcontact vertical , Qcontact horizontal and the target
direction Qtarget direction are set based on the positionand direction, respectively, of the target character orobject. If an adjective is used in the input text (e.g.
“top of the table”, “under the table”) and the object
has the corresponding part (see Section 3), the position of the corresponding part assigned to the object
is used as the contact position. If there is no adjective,the default position specified for the object is used. The
instrument object that is indicated in the input text is
also set to the query frame.

5.2 Temporal Constraints
Temporal constraints are extracted from input texts.
The types of temporal constraint are serial execution
or synchronized execution between two verbs. A serial
execution constraint has the execution order of two motions. A synchronized execution constraint has relative
execution timing. Temporal constraints are generated
from a syntax tree as follows:
1. For all pairs of sequential verbs in the input text, serial execution constraints are assigned. For example,
when the input text “Jack walks in the room. Neostands up.” is given to the system, a serial executionsection constraint (Jack, walk) to (Neo, stands up)
is generated.
2. When a word that indicates a reverse order exists in
the input text (e.g., “after”), the order of the serial
execution constraint is reversed. If a serial execution
constraint is already created, the old constraint isoverridden. For example, when the input text “Jackwalks in the room after Neo stands up.” is given
to the system, a serial execution constraint (Neo,stands up) to (Jack, walk) is generated.
3. When a word that indicates synchronization exists
in the input text (e.g., “at the same time” or “while”),a synchronized execution constraint is added. If there
is a conflicting constraint, it is overridden. For example, when the input text “Jack walks in the room.
At the same time, Neo stands up.” is given to thesystem, a synchronized execution constraint (Neo,stands up) and (Jack, walk) is generated. The relative timings between two motions are set to zero sothat they start at the same time.
4. When the motions of two characters are cooperative
motions and they include contact with each other, a
synchronized execution constraint is added and therelative execution timings of the two motions are
determined based on their contact information (Section 4.3). For example, when the input text “Jackhits Neo. Neo falls” is given to the system, a synchronized execution constraint (Jack, hit) and (Neo,fall) is generated. At this point, the relative timings
are not set. They will be set based on the contact
times in the searched motion data, when the motions are searched later.

5.3 Adjective and Pronouns
Sometimes a character or an object is referred to by
a combination of adjectives and a noun instead of its
name. In this case, the system has to determine towhich character or object the expression refers. Moreover, when a pronoun (e.g., “he”, “she”, “it”, “they”)
is used in the input text, the system has to determine
to which character or object the pronoun refers. This
process can be difficult especially when ambiguous or
euphemistic expressions are used. However, since oursystem is meant to take simple script-like texts, we handle these problems using the following method.
Each character and object in our database has alist of adjectives, nouns, and pronouns by which thecharacter can be referred to. For example, a male soldier character would have “he”, “they”, “man”, “guy”,“soldier”, etc. as its list of adjectives and pronouns.
The list is inherited from the base character. If the adjective, noun, or pronoun appearing in the input text
represents a single character or object (e.g., “he”, “thesoldier”, “it”), the system searches for a character oran object that matches the words.
However, if the words are ambiguous, meaning that
there are multiple characters or objects matching the
given adjective, noun, or pronoun in the scene, the system has to choose an appropriate character or object.
Basically, if the noun or pronoun represents a characteror an object, this should be mentioned in the previous sentences. A character can be the agent Qagent refor the target Qtarget ref of the query frame, while an
object (e.g., “it”) can be the target Qtarget ref or the
instrument Qinstrument ref thereof. By using this constraint, the noun or pronoun is determined as follows.

9

1. If one of two items is clearly mentioned in the input
text, the other should be a different character or object. For example, if the input text is “Neo comes toJack. Jack gives him a book”, the word “him” cannot represent Jack since Jack is already the agent
Qagent ref of the query frame of the second sentence.
Therefore, in this case, the system searches for characters in the previous sentence and uses the othercharacter Neo as the target Qtarget ref .
2. If both of the items for characters are pronouns,
the agent of the previous sentence is also used as
the agent Qagent ref . For example, if the input text
is “Neo comes to Jack. He gives him a book”, thesystem decides that the first “he” in the second sentence represents Neo. In the same way, if both theitems for objects are pronouns, although this is notcommon, the target of the previous sentence is usedas the current target and the instrument of the previous sentence is used as the current instrument.
If the noun or pronoun represents multiple characters (e.g., “they”, “soldiers”), the sentence should be
represented by multiple motions. Therefore, in such acase, multiple query frames are generated. For example,
if the input text is “Neo hits the soldiers” and there are
two soldiers A and B in the scene, two query frames
are generated with all items except the target character
the same. The same rule is applied when an item of thequery frame represents multiple characters (e.g., “Neoand Jack walk.”). However, if a motion frame matching the motion name has a target direction, but not acontact position, the center position of all characters referred to is used as the target direction, instead of generating multiple query frames. For example, with the
input “Neo shoots the soldiers”, the “shooting” motion
frame has only a target direction, and a query frame
whose target position is the center of the soldiers is
generated.

5.4 Infinitives and Gerunds
Infinitives and gerunds are often used with a verb. In
this case, the system generates appropriate query frames
and temporal constraints depending on the verb.
– If the verb is “do”, “perform”, etc. (e.g., “Neo performs dancing“), the infinitive or the gerund is represented as the motion and is used to generate acorresponding query frame. In this case, the verb is
not involved in the query frame.
– If the verb is “start”, “try”, etc., the infinitive or thegerund is used to generate a corresponding query
frame in the same way. However, in this case, thenext event is considered to happen before the motion finishes. Therefore, a temporal constraint is generated to execute the next motion just after thismotion starts.
– If the verb is “repeat”, “keep”, etc., the infinitiveor gerund is used to generate a corresponding query
frame. The motion is repeated before the next motion starts, and is therefore, specified in the query
frame. This information is used for motion scheduling to duplicate a motion when it can be executed
more than once.
– It the verb does not fall into any of the above cases,
both the verb and the infinitive or gerund are represented as motions (e.g., “Neo walks waving toJack”). In this case, multiple query frames are generated. In addition, temporal constraints are generated to execute all motions at the same time.
The system uses a dictionary of pairs consisting of
the verb and the corresponding method, to determine
which method should be applied.
In addition, if a gerund exists on its own in the
input text (“Neo walks to the door, waving to Jack”), a
query frame is generated for the gerund and a temporalconstraint is generated to execute the gerund and theverb in the sentence at the same time. In this case, theagent of the verb becomes the agent of the gerund aswell.

5.5 Locomotive Motions
Locomotive motions such as walking and running require special care, because the target position and pathvary depending on the situation and appropriate motions should be generated instead of simply executing
a motion in the database. How to generate locomotive
motions is explained in Section 6.3. In this section, we
explain how to handle locomotive motions in natural
language analysis.
As discussed in Section 8, natural language is notsuited to specifying the locomotion path. Therefore,
our system currently does not handle it and only determines the target position of locomotive motions. We
categorize locomotive motions into the following types
depending on how the target position is handled.
– Moving to a target position. If the verb is a locomotive motion (e.g., “walk”, “run”, “go”, etc.) anda target position is explicitly specified in the input text (e.g., “Neo walks to the door.”), the query
frame includes the target position Qcontact virtical ,Qcontact horizontal and a flag indicating that this isa locomotive motion.

10– Entering and leaving. Sometimes the target position is not specified in the input text. In this case, if
the verb is a specific verb, the locomotive motion ishandled as an entering or leaving motion. For example, if the verb is “leave”, “walk away”, “disappear”,
etc., the verb is handled as a leaving motion and theleaving position specified in the scene information isused as the target position. The query frame also includes a flag indicating locomotive motion.
– Simple walking. If a target position is not specified and the verb represents a locomotive motion
excluding an entering or leaving motion (e.g., “Neowalks.”), a walking motion is simply executed. In
this case, the query frame does not include a flag
indicating locomotive motion. This query frame ishandled in the same way as the other query frames.
As a result, a motion of walking forward from thecharacter’s current position is selected and executed.

5.6 Adverbs
Adverbs are handled in different ways depending on
the word. As explained in Section 5.2, if the adverb
represents temporal information, an appropriate temporal constraint is generated. If the adverb represents
the frequency or timing of executing a verb, the adverb
is handled in the same way as infinitives and gerunds
in Section 5.4. For example, if an adverb such as “repeatedly”, “twice”, etc. is specified, the third option in
Section 5.4 is applied. The system has a dictionary ofadverbs for these cases. If the adverb is not found in
the dictionary, it is assigned to a query frame to search
for an appropriate motion as explained in Section 5.1.

6 Motion Search
In this section, we explain how to search for an appropriate motion for each verb in the input text. Handling
multiple verbs and motions is dealt with in the next
section. A query frame is generated for each verb as
explained in the previous section. Based on the query
frame, a motion is selected from the database.

6.1 Evaluation of Motion Frame
A motion frame that best matches the query frame issearched for in the database. This search is performedin three steps.
In the first step, all candidate motion frames inwhich the motion name and agent match the query
frame are selected from the database. All motion frameswith the agent character or its base characters are potential candidates.
In the second step, the motion frames whose items
do not match the query frame are excluded as candidates. If the query frame has a target Qtarget ref , or
Qtarget params and/or an instrument Qinstrument ref ,or Qinstrument params but the motion frame does not,then it is excluded. Moreover, if a motion frame hastarget parameters, instrument parameters, or the vertical contact position, and the values of the query frame
exceed the specified ranges, then that motion frame isalso excluded.
In the third step, all candidate motion frames areevaluated based on the similarity between the motion
frame and the query frame items using the following
equation:

E = w0 R(Mtarget params , Qtarget params )+w1 R(Minstrument params , Qinstrument params )+w2 D(Mcontact vertical , Qcontact vertical )+w3 D(Mcontact horizontal , Qcontact horizontal )+w4 D(Mtarget direction , Qtarget direction )+w5 F (Minitial posture f lag , Qinitial posture f lag )+w6 A(Madverb strings , Qadverb strings )+w7 H(Magent ref , Qagent ref )(1)

where R(M, Q), D(M, Q), F (M, Q), A(M, Q), H(M, Q)
are the functions that compute normalized distance(0.0∼1.0) between size and weight parameters, contact
positions, posture flags, adverbs, and hierarchical positions, respectively. The distances between the size andweight range of the motion frame and the object size
and weight of the query frame are computed so thatthe distance becomes zero when the values are at thecenter of the range and the distance becomes one whenthe values are at the edge of the range. The distance
between posture flags is computed in such a way thatthe distance is zero when they match and otherwise
the distance is one. The distance between adverbs iscomputed so that the distance is zero when there is atleast one pair of matching adverb between the motion
frame and query frame and otherwise the distance isone. The distance between hierarchical positions of thecharacters is computed from the number of inheritances
between them (see Fig 5). The candidate motion frame
whose evaluation is the smallest will be selected andused for animation. w0 ∼ w7 are weight parameters.
They can be set for each motion frame in the case thatsome items are important for the motion. In our current
experiments we used 1.0 for all weights on all motions.

11

6.2 Motion Modification
The motion clip of the selected motion frame is used foranimation. However, even if the closest motion frame isselected, the contact position may not exactly match
the query frame. In that case, the motion clip is modified using inverse kinematics. The posture of the character during the motion is modified so that the contact
position of the end-effector (e.g., hand) matches thetarget position in the query frame.
When the character is far from the target, changing the end-effector position is not enough. In addition,
when the character executes the selected motion it may
need to first take an instrumental object or change its
posture (e.g., standing up). These cases are handled byadding automatic motions before the selected motion
instead of modifying the selected motion. Automatic
motions are explained in Section 7.2 .

6.3 Locomotive Motion
When a query frame indicates a locomotive motion (seeSection 5.5), appropriate motion enabling the character to move to the target position must be generated.
Several methods, such as [16], have been developed togenerate walking motions. In our system, the character should not only walk, but also turn and step, inorder to move to the appropriate position and direction. Therefore, we generate locomotive motions basedon the target position and/or target direction according to the steps below using the set of motion data thatthe character has.
1. If the target direction is specified and the target position is not, an appropriate turning motion is generated. An appropriate motion based on the target
direction is selected from the motions with ‘turn’as their motion name in the database. If only thetarget direction is specified, the process stops here.
2. If the target position is specified and it is not infront of the character, a turning motion is added in
the same way as in the first step so that the agent
faces the target position.
3. If the target position is within one step, a stepping
motion is added in the same way as the turning motion. The motion is selected from all ‘step’ motions.
4. If the target position is more than one step in the
distance, a walking motion is added. The walking
motion is repeated until the agent reaches the target position. The step length in each walking cycle
is adjusted so that the walking cycle ends at thetarget position. The motion is selected from ‘walk’
motions. Currently, our system generates a straightpath to the target position even if there are obstacles.
5. If the target direction is specified and it does notmatch the character’s direction at the end point of
the walking motion, a turning motion is once again
added.
As explained above, the system uses the “turn”,“step” and “walk” motions that the character has. Currently the system selects an appropriate motion andmodifies it if necessary. Alternatively, motion blending
can be used to generate more appropriate motions byusing multiple motions [16][18].

7 Motion SchedulingIn this section, we explain how our system handles multiple motions from an input text. Basically, the systemsearches for a motion for each verb in the input text.
However, in order to make an animation, the execution
timing of each motion must also be determined. Moreover, the continuity of motions should be considered.
For example, when a character makes contact with anobject in the scene, the character must first move close
to the object. Our system takes care of this kind ofcontinuity of motions.
When multiple characters perform multiple motions
the motions should be scheduled. However, an exact
execution time for each motion is not usually specified
in the input text. In order to determine the motion
schedule, we need information about the motions suchas duration and contact information.
Our motion schedule works as follows. First, temporal constraints are extracted from input texts in addition to query frames (Section 5). Second, query frames
are roughly scheduled based on the temporal constraints(Section 7.1). Note that at this point, only process orders of query frames are determined. Finally, by searching for a motion frame that matches each query frame inorder of process, the execution timing of each motion
is determined. When automatic motions are required
to be executed before a motion, they are added incrementally (Section 7.2). By repeating this process for
all query frames, the motion clips and their execution
timings are determined.

7.1 Scheduling Query Frames
Based on temporal constraints, the query frames arescheduled roughly at first. After that, the process orderof all query frames (verbs) is determined. For motions
that have a synchronized execution constraint, their

12

process orders are temporarily set as one of them being
processed first. The exact timings of all query frames
are determined in the process order.
For each query frame, a motion clip is searched fromthe database as explained in Section 6.1. Before searching each motion, the scene condition is set to the time
when the motion is executed because the selected motion may change depending on the position of the character or object that the motion involves. The execution
timing of the motion is determined based on the duration of the selected motion. The next motion is started
just after the previous motion is finished if they havea serial execution constraint. If they have a synchronized executing constraint, their execution timings are
determined based on the contact timings of the selectedmotions.
This process is repeated from the first motion to
the last. When multiple query frames are synchronizedbased on the temporal constraints, the motions for all
query frames are searched and their execution timings
are delayed until all constraints are satisfied.

the motion frame has a contact position, an appropriate locomotive motion is generated so that the character approaches the right point. If the motion frame
only has a direction (e.g., “shooting toward the target”
motion), the character merely turns without walking.
As explained in Section 6.3, our current system has nopath planning; the character merely moves in a straightline to the target position.

7.2 Automatic Motions

For motion searches, if there is no candidate motion
whose initial posture matches the terminal posture of
the previous motion (i.e., the initial posture of a query
frame does not match any of the candidate motion
frames), a changing posture motion such as standing
up is added. In this case, all motions that include astate change will be candidate motions.

During the motion scheduling and motion search, asearched motion can sometimes not be executed. Inthat case, automatic motions are generated and added
before the searched motion. As explained earlier, thepurpose of our system is to reuse motion data withoutcomplex motion planning which may require additional
programming for each motion. Therefore, our systemdeals with minimum automatic motions. The additionalmotions are also selected from the database. Therefore,
each character is easily customized by adding specifickinds of motion to the database without adding any
rules or modules.
7.2.1 Locomotive motion
If a motion includes interaction with another characteror an object in the scene (i.e., a query frame contains
a target object or character), the character has to bein the right place to make contact with the object orcharacter. If not, the system automatically adds locomotive motions for the character to move to the rightplace and to face the right direction.
If the motion frame has a contact position and target direction (e.g., “sitting on a chair” motion should
be executed in the right position and direction to thechair), an appropriate locomotive motion is generated
so that the character approaches the right point andturns in the right direction. The method for generating
locomotive motions explained in Section 6.3 is used. If

7.2.2 Taking an instrument
When a character uses an instrument in a motion (i.e.,a query frame contains an instrument and the character does not hold it), the character must pick up the
instrument object before they use it. When a motion
to take the instrument is not explicit in the input text,a ‘take’ motion is selected from the database. Whenthe character is away from the instrument, locomotive
motions are also added before the taking motion.
7.2.3 Changing posture

7.2.4 Cooperative motion
As explained in Section 4.3, when a motion involves interaction with another character, a cooperative motionof the other character follows. When a selected motion
frame has cooperative motions and any of them are notindicated in the input text, the default cooperative motion and a temporal constraint of the motion frame areautomatically added.

8 Experiment and Discussion
We have implemented our method and motion database.
Currently, the system has six characters as shown in
Fig. 5 and about 50 motions that are collected from acommercially available motion capture library. We havetested our system with some short sentences and foundthat an appropriate motion was selected from each sentence even though the same verb is used in different sentences. An example of the generated animation is available from the author’s web site (http://www.cg.ces.
kyutech.ac.jp/research/modb/index.html).

13

To evaluate our framework, we tested it with a published movie script (The Matrix, 1999). Because our
motion database does not yet have enough data, wechecked whether our methods could handle the descriptions in the movie script and output appropriate query
frames. There were about 830 actions (verbs) in thescript. We found that about 87% of these were processed by our system without any problems. However,
4% were complex expressions that are difficult to handle
using simple rules, such as a sentence with the subject
being a character’s body (e.g., “His elbow hits the enemy.”, “His body jumps.”), vague representation (e.g.,“he stares into the darkness”), indirect expression (e.g.,“He has no answer.”), and ambiguous nouns or pronouns. 4% were verbs that cannot be represented bya motion including non-action or state verbs explainedin Section 4.2, such as “He feels that ∼”, a verb representing a result of a motion such as “miss” in “heshoots her and misses”. 5% were verbs representing initial states in the scene but not actions (e.g., “they aredead”, “he stands in the room”). As discussed later, a
non-text-based interface is suitable for specifying initial states or positions of locomotion. According to theabove results, 9% of the verbs in the sample script wereactually verbs that cannot be represented as motions.
This shows that 95% of verbs that can be represented
as motion were handled by our methods. Although it
is possible to extend our semantic analysis to support
more complex expressions, this will require a great dealof knowledge and rules, which is contrary to the aims ofthis research. If a complex expression cannot be handled
by the system, the user should rephrase it as a plain expression rather than adding more knowledge and rules
to the system.
However, even if an animation is generated from agiven text, since there is limited information in an input
text, a user may not be satisfied with the motions thatare found in the database. Moreover, since we use a simple method for generating locomotive motion, motion
modification, and motion synthesis, a user may not besatisfied with the synthesized animation. To evaluate
the effectiveness of our system, we intend conducting
a practical user study in a future work. Improvementof motion generation and the external motion synthesis
system [20] is also a future work.
Our current system cannot handle object motions.
However, as they are also important for animation, it
is easily possible to extend our system to handle them,as they tend to be simpler than human motions.
The fundamental principle of our framework is tomake use of motion data without requiring any additional motion specific rules. Currently, our system does
not support high-level motion planning such as auto-

matically dividing complex motion into small motions
or path planning with object avoidance. Because weuse simple rules for automatic locomotion, the resulting animations are not so natural. This can be solved
by adding more motion data and some sophisticatedmodules that generate new motion from a number ofmotion data sources such as [17][18].
Our system supposes that scene information, suchas the positions of objects and characters, is provided by
the user. The existing text-to-scene system [10] can be
integrated with our system. However, specifying the positions using natural language can be harder than using
a conventional mouse-based interface. So can specifying
locomotion path. From a practical viewpoint, a hybridof a text-based interface and a conventional interface
might be more useful.
With our current system, if the user is not satisfied
with or wants to change an output motion, they mustchange the input text and they cannot change the output motions directly. To address this, we are going todevelop a natural language-based motion editing interface with which a user can change generated motions
interactively by giving instructions to agents, as real
directors do with actors.
9 Conclusion
We have proposed an animation system that generates
animation from natural language text such as movie
scripts or stories. Our future work includes the expansion of both the system and the motion database. Currently, creating animations is very difficult, especiallyfor nonprofessionals. We believe that our system will
alleviate this and provide many creators with a meansof expressing their stories as animation.
References
1. Masaki Oshita, “Generating animation from natural language
texts and framework of motion database”, In Proc. of International Conference on Cyberworlds 2009, pp. 146-153, Bradford,
UK, September 2009.
2. Terry Winograd. Understanding Natural Language. Academic
Press, 1972.
3. N. Badler, R. Bindiganavale, J. Allbeck, W. Schuler, L. Zhao,
and M. Palmer. “Parameterized action representation for virtual human agents”, In Embodied Conversational Agents, pp.
256-284, 2000.
4. R. Bindiganavale, W. Schuler, J. Allbeck, N. Badler, A. Joshi,
and M. Palmer. “Dynamically altering agent behaviors using
natural language instructions”, In Proc. of Autonomous Agents
2000, pp. 293-300, 2000.
5. Takenobu Tokunaga, Kotaro Funakoshi, and Hozumi Tanaka.
“K2: animated agents that understand speech commands andperform actions”, In Proc. of 8th Pacific Rim International
Conference on Artificial Intelligence 2004, pp. 635-643, 2004.

14
6. Charles J Fillmore. The case for case. In Universals in Linguistic Theory, pp. 1-88, 1968.
7. Ruqian Lu, Songmao Zhan, Automatic Generation of Computer Animation: Using AI for Movie Animation, Springer,
2002.
8. Kaoru Sumi and Mizue Nagata. “Animated storytelling system via text”, In Proc. of International Conference on Advancesin Computer Entertainment Technology, 2006.
9. Hiromi Baba, Tsukasa Noma, and Naoyuki Okada. “Visualization of temporal and spatial information in natural language
descriptions”, Transaction on Information and Systems, E79D(5), pp. 591-599, 1996.
10. Bob Coyne and Richard Sproat. “Wordseye: an automatic
text-to-scene conversion system”, In Proc. of SIGGRAPH 2001,
pp. 487-496, 2000.
11. Sergey Levine, Christian Theobalt, Vladlen Koltun. “Realtime prosody-driven synthesis of body language”, ACM Transactions on Graphics (In Proc. of ACM SIGGRAPH Asia 2009),
Vol. 28, Issue 5, 2009. (to appear)
12. Ken Perlin, and Athomas Goldberg, “Improv: a system for
scripting interactive actors in virtual worlds”, In Proc. of SIGGRAPH ’96 Proceedings, pp. 205-216, 1996.
13. Matthew J. Conway. Alice: easy-to-learn 3D scripting for
novices, PhD Dissertation, University of Virginia, 1997.
14. Masaki Hayashi, Hirotada Ueda, Tsuneya Kurihara, Michiaki Yasumura, “TVML (TV program Making Language) - automatic TV program generation from text-based script –”, In
Proc. of Imagina ’99, pp. 84-89, 1999.
15. Hyunju Shim, Bo Gyeong Kang, “CAMEO - camera, audio and motion with emotion orchestration for immersive cinematography”, In Proc. of International Conference on Advancesin Computer Entertainment Technology (ACE) 2008, pp. 115118, 2008.
16. Sang Il Park, Hyun Joon Shin, Sung Yong Shin, “On-line
locomotion generation based on motion blending”, In Proc. of
ACM SIGGRAPH Symposium on Computer Animation 2002,
pp. 105-111, 2002.
17. C. Rose, M. F. Cohen, and B. Bodenheimer. “Verbs andadverbs: Multidimensional motion interpolation”, IEEE Computer Graphics and Applications, vol. 18, no. 5, pp. 32-40, 1998.
18. Lucas Kovar and Michael Gleicher. “Automated extraction
and parameterization of motions in large data sets”, ACM
Transactions on Graphics, vol. 23, no. 3, pp. 559-568, 2004.
19. Dan Klein and Christopher D. Manning. “Fast exact inference with a factored model for natural language parsing”, In
Advances in Neural Information Processing Systems 15 (NIPS
2002), pp. 3-10, 2003.
20. Masaki Oshita. “Smart motion synthesis”, Computer Graphics Forum, vol. 27, no. 7, pp. 1909-1918, 2008.