Authoring and use systems for sound synchronized animation

A general purpose computer, such as a personal computer, is programmed for sound-synchronized random access and display of synthesized actors ("synactors") on a frame-by-frame basis. The interface between a user and the animation system is defined as a stage or acting metaphor. The user interface provides the capability to create files defining individually accessible synactors representing real or imaginary persons, animated characters and objects or scenes which can be programmed to perform speech synchronized action. Synactor speech is provided by well-known speech synthesis techniques or, alternatively, by inputting speech samples and communication characteristics to define a digital model of the speech and related animation for a particular synactor. A synactor is defined as combination of sixteen predefined images; eight images to be synchronized with speech and eight images to provide additional animated expression. Once created, a synactor may be manipulated similarly to a file or document in any application. Once created, a synactor is controlled with scripts defined and edited by a user via the user interface.

BACKGROUND OF THE INVENTION 
The present invention relates generally to computerized animation methods 
and, more specifically to a method and apparatus for creation and control 
of random access sound-synchronized talking synthetic actors and animated 
characters. 
It is well-known in the prior art to provide video entertainment or 
teaching tools employing time synchronized sequences of pre-recorded video 
and audio. The prior art is best exemplified by tracing the history of the 
motion picture and entertainment industry from the development of the 
"talkies" to the recent development of viewer interactive movies. 
In the late nineteenth century the first practical motion pictures 
comprising pre-recorded sequential frames projected onto a screen at 20 to 
30 frames per second to give the effect of motion were developed. In the 
1920's techniques to synchronize a pre-recorded audio sequence or sound 
track with the motion picture were developed. In the 1930's animation 
techniques were developed to produce hand drawn cartoon animations 
including animated figures having lip movements synchronized with an 
accompanying pre-recorded soundtrack. With the advent of computers, more 
and more effort has been channeled towards the development of computer 
generated video and speech including electronic devices to synthesize 
human speech and speech recognition systems. 
In a paper entitled "KARMA: A system for Storyboard Animation" authored by 
F. Gracer and M. W. Blasgen, IBM Research Report RC 3052, dated Sep. 21, 
1970, an interactive computer graphics program which automatically 
produces the intermediate frames between a beginning and ending frame is 
disclosed. The intermediate frames are calculated using linear 
interpolation techniques and then produced on a plotter. In a paper 
entitled "Method for Computer Animation of Lip Movements", IBM Technical 
Disclosure Bulletin, Vol. 14 No. 10 Mar., 1972, pages 5039, 3040, J. D. 
Bagley and F. Gracer disclosed a technique for computer generated lip 
animation for use in a computer animation system. A speech-processing 
system converts a lexical presentation of a script into a string of 
phonemes and matches it with an input stream of corresponding live speech 
to produce timing data. A computer animation system, such as that 
described hereinabove, given the visual data for each speech sound, 
generates intermediate frames to provide a smooth transition from one 
visual image to the next to produce smooth animation. Finally the timing 
data is utilized to correlate the phonetic string with the visual images 
to produce accurately timed sequences of visually correlated speech 
events. 
Recent developments in the motion picture and entertainment industry relate 
to active viewer participation as exemplified by video arcade games and 
branching movies. U.S. Pat. Nos. 4,305,131; 4,333,152; 4,445,187 and 
4,569,026 relate to remote-controlled video disc devices providing 
branching movies in which the viewer may actively influence the course of 
a movie or video game story. U.S. Pat. No. 4,569,026 entitled "TV Movies 
That Talk Back" issued on Feb. 4, 1986 to Robert M. Best discloses a video 
game entertainment system by which one or more human viewers may vocally 
or manually influence the course of a video game story or movie and 
conduct a simulated two-way voice conversation with characters in the game 
or movie. The system comprises a special-purpose microcomputer coupled to 
a conventional television receiver and a random-access videodisc reader 
which includes automatic track seeking and tracking means. One or more 
hand-held input devices each including a microphone and visual display are 
also coupled to the microcomputer. The microcomputer controls retrieval of 
information from the videodisc and processes viewers' commands input 
either vocally or manually through the input devices and provides audio 
and video data to the television receiver for display. At frequent branch 
points in the game, a host of predetermined choices and responses are 
presented to the viewer. The viewer may respond using representative code 
words either vocally or manually or a combination of both. In response to 
the viewer's choice, the microprocessor manipulates pre-recorded video and 
audio sequences to present a selected scene or course of action and 
dialogue. 
In a paper entitled "Soft Machine: A Personable Interface", "Graphics 
Interface '84", John Lewis and Patrick Purcell disclose a system which 
simulates spoken conversation between a user and an electronic 
conversational partner. An animated person-likeness "speaks" with a speech 
synthesizer and "listens" with a speech recognition device. The audio 
output of the speech synthesizer is simultaneously coupled to a speaker 
and to a separate real-time format-tracking speech processor computer to 
be analyzed to provide timing data for lip synchronization and limited 
expression and head movements. A set of pre-recorded visual images 
depicting lip, eye and head positions are properly sequenced so that the 
animated person-likeness "speaks" or "listens". The output of the speech 
recognition device is matched against pre-recorded patterns until a match 
is found. Once a match is found, one of several pre-recorded responses is 
either spoken or executed by the animated person-likeness. 
Both J. D. Bagley et al and John Lewis et al require a separate 
format-tracking speech processor computer to analyze the audio signal to 
provide real-time data to determine which visual image or images should be 
presented to the user. The requirement for this additional computer adds 
cost and complexity to the system and introduces an additional source of 
error. 
SUMMARY OF THE INVENTION 
The present invention provides a method and apparatus for a random access 
user interface referred to as hyperanimator, which enables a user to 
create and control animated lip-synchronized images or objects utilizing a 
personal computer. The present invention may be utilized as a general 
purpose learning tool, interface device between a user and a computer, in 
video games, in motion pictures and in commercial applications such as 
advertising, information kiosks and telecommunications. Utilizing a 
real-time random-access interface driver (RAVE) together with a 
descriptive authoring language called RAVEL (real-time random-access 
animation and vivification engine language), synthesized actors, 
hereinafter referred to as "synactors", representing real or imaginary 
persons and animated characters, objects or scenes can be created and 
programmed to perform actions including speech which are not sequentially 
pre-stored records of previously enacted events. Animation and sound 
synchronization are produced automatically and in real-time. 
The communications patterns--the sounds and visual images of a real or 
imaginary person or of an animated character associated with those 
sounds--are input to the system and decomposed into constituent parts to 
produce fragmentary images and sounds. Alternatively, or in conjunction 
with this, well known speech synthesis methods may also be employed to 
provide the audio. That set of communications characteristics is then 
utilized to define a digital model of the motions and sounds of a 
particular synactor or animated character. A synactor that represents the 
particular person or animated character is defined by a RAVEL program 
containing the coded instructions for dynamically accessing and combining 
the video and audio characteristics to produce real-time sound and video 
coordinated presentations of the language patterns and other behavior 
characteristics associated with that person or animated character. The 
synactor can then perform actions and read or say words or sentences which 
were not prerecorded actions of the person or character that the synactor 
models. Utilizing these techniques, a synactor may be defined to portray a 
famous person or other character, a member of one's family or a friend or 
even oneself. 
In the preferred embodiment, hyperanimator, a general purpose system for 
random access and display of synactor images on a frame-by-frame basis 
that is organized and synchronized with sound is provided. Utilizing the 
hyperanimator system, animation and sound synchronization of a synactor is 
produced automatically and in real time. Each synactor is made up of 
sixteen images, eight devoted to speaking and eight to animated 
expressions. 
The eight speaking images correspond to distinct speech articulations and 
are sufficient to create realistic synthetic speaking synactors. The 
remaining eight images allow the synactor to display life-like 
expressions. Smiles, frowns and head turns can all be incorporated into 
the synactor's appearance. 
The hyperanimator system provides the capability to use both synthetic 
speech and/or digitized recording to provide the speech for the synactors. 
Speech synthesizers can provide unlimited vocabulary while utilizing very 
little memory. To make a synactor speak, the text to be spoken is typed or 
otherwise input to the system. Then the text is first broken down into its 
phonetic components. Then the sound corresponding to each component is 
generated through a speaker as an image of the synactor corresponding to 
that component is simultaneously presented on the display device. 
Digitized recording provides digital data representing actual recorded 
sounds which can be utilized in a computer system. Utilizing a 
"synchronization lab" defined by the hyperanimator system, a synactor can 
speak with any digitized sound or voice that is desired. 
The interface between the user and the hyperanimator system is defined as a 
stage or acting metaphor. The hyperanimator system allows the user to 
shift or navigate between a number of display screens or cards to create 
and edit synactor files. While other paradigms are possible, this one 
works well and allows relatively inexperienced users to understand and 
operate the hyperanimator system to create, edit and work with the 
synactors. 
The dressing room is where synactors are created and edited and is where 
users and synactors spend most of their time. The dressing room comprises 
16 cards, 1 for each of the synactor images describing a synactor. Buttons 
are provided on each card to allow the user to navigate between the cards 
by pressing or clicking on a button with a mouse or other input device. 
Within the dressing room, the image of the synactor is placed in a common 
area named the Synactor Easel. Utilizing separate utilities such as "paint 
tools" or "face clip art", the user can create and edit the synactor. With 
a paint tool, a synactor may be drawn from scratch or, with clip art, a 
synactor may be created by copying and "pasting" eyes, ears, noses and 
even mouths selected from prestored sets of the different features. 
Once the synactor has been created or built in the dressing room, the user 
can transfer the synactor to a stage screen where the lip synchronization 
and animation of the actor may be observed. The stage screen includes a 
text field wherein a user can enter text and watch the synactor speak. If 
the synactor thus created needs additional work, the user can return the 
synactor to the dressing room for touchup. If the user is satisfied with 
the synactor, the synactor can be then saved to memory for future use. 
In the hyperanimator system, the synactor file is manipulated like a 
document in any application. Copying, editing (transferring a synactor 
file to the dressing room) and deleting actors from memory is accomplished 
in the casting call screen. The casting call screen displays a stagehand 
clipboard and provides buttons for manipulating the synactor files. 
Copying and deleting sound resources comprising digitized sounds is 
accomplished in the sound booth screen. The digitized sound resources are 
synchronized with the image of the synactor in the screen representing the 
hyperanimator speech synchronization lab. The speech sync lab examines the 
sound and automatically creates a phonetic string which is used to create 
the animation and sound synchronization of the synactor. The speech sync 
lab generates a command called a RECITE command which tells the RAVE 
driver which sound resource to use and the phonetic string with associated 
timing values which produces the desired animation. The speech sync lab 
also provides for testing and refinement of the animation. If the 
synchronization process is not correct, the user can modify the RECITE 
command manually. 
The above described functions and screens are tied together and accessed 
essentially from a menu screen. The menu screen contains six buttons 
allowing a user easy navigation through the screens to the hyperanimator 
system features. At the center of the menu screen is displayed a synactor 
called the Hyperanimator Navigator who serves a guide for a user through 
the hyperanimator system. The RAVE system is responsible for the animation 
and sound synchronization of the synactors. RAVEL defines and describes 
the synactor while the RAVE scripting language is an active language which 
controls the synactor after it is created by a user. RAVE scripting 
language commands enable a programmer to control the RAVE for an 
application program created by the programmer utilizing a desired 
programming system. Utilizing facilities provided in the programming 
system to call external functions, the programmer invokes the RAVE and 
passes RAVE scripting language commands as parameters to it. The RAVE 
script command controller 43 interprets these commands to control the 
synactor. 
Once a synactor is created, it is controlled in a program by scripts 
through the RAVE scripting language level. All of the onscreen animation 
is controlled by scripts in the host system through the RAVE scripting 
language. Various subroutines called external commands ("XCMD") and 
external functions ("XFCN") are utilized to perform functions not 
available in the host language, for example creating synactors from the 
dressing room. The RAVE XCMD processes information between the scripts and 
the RAVE driver. Fifteen separate commands are utilized to enable users to 
open, close, move, hide, show and cause the synactor to speak. A program 
may have these commands built in, selected among or generated by the RAVE 
driver itself at runtime. 
The hyperanimator system of the present invention is user friendly and 
easily understood by inexperienced users. It provides a user with the 
capability to create animated talking agents which can provide an 
interface between people and computers.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
Referring now to FIG. 1, in one preferred embodiment of the present 
invention, a special purpose microcomputer comprises a program controlled 
microprocessor 10 (a Motorola MC68000 is suitable for this purpose), 
random-access memory (RAM) 20, readonly only memory (ROM) 11, disc drive 
13, video and audio input devices 7 and 9,1 user input devices such as 
keyboard 15 or other input devices 17 and output devices such as video 
display 19 and audio output device 25. RAM 20 is divided into four blocks 
which are shared by the microprocessor 10 and the various input and output 
devices. 
The video output device 19 may be any visual output device such as a 
conventional television set or the CRT for a personal computer. The video 
output 19 and video generation 18 circuitry are controlled by the 
microprocessor 10 and share display RAM buffer space 22 to store and 
access memory mapped video. The video generation circuits also provide a 
60 Hz timing signal interrupt to the microprocessor 10. 
Also sharing the audio RAM buffer space 23 with the microprocessor 10 is 
the audio generation circuitry 26 which drives the audio output device 25. 
Audio output device 25 may be a speaker or some other type of audio 
transducer such as a vibrator to transmit to the hearing impaired. 
Disc controller 12 shares the disc RAM 21 with the microprocessor 10 and 
provides control reading from and writing to a suitable non-volatile mass 
storage medium, such as floppy disc drive 13, for long-term storing of 
synactors that have been created using the hyperanimator system and to 
allow transfer of synactor resources between machines. 
Input controller 16 for the keyboard 15 and other input devices 17 is 
coupled to microprocessor 10 and also shares disc RAM 21 with the disc 
controller 12. This purpose may be served by a Synertek SY6522 Versatile 
Interface Adaptor. Input controller 16 also coordinates certain tasks 
among the various controllers and other microprocessor support circuitry 
(not shown). A pointing input device 17 such as a mouse or light pen is 
the preferred input device because it allows maximum interaction by the 
user. Keyboard 15 is an optional input device in the preferred embodiment, 
but in other embodiments may function as the pointing device, or be 
utilized by an instructor or programmer to create or modify instructional 
programs or set other adjustable parameters of the system. Other pointing 
and control input devices such as a joy stick, a finger tip (in the case 
of a touch screen) or an eyemotion sensor are also suitable. 
RAM 24 is the working memory of microprocessor 10. The RAM 24 contains the 
system and applications programs other information used by the 
microprocessor 10. Microprocessor 10 also accesses ROM 11 which is the 
system's permanent read-only memory. ROM 11 contains the operational 
routines and subroutines required by the microprocessor 10 operating 
system, such as the routines to facilitate disc and other device I/0, 
graphics primitives and real time task management, etc. These routines may 
be additionally supported by extensions and patches in RAM 24 and on disc. 
Controller 5 is a serial communications controller such as a Zilog Z8530 
SCC chip. Digitized samples of video and audio may be input into the 
system in this manner to provide characteristics for the talking heads and 
synthesized speech. Digitizer 8 comprises an audio digitizer and a video 
digitizer coupled to the video and audio inputs 7 and 9, respectively. 
Standard microphones, videocameras and VCRs will serve as input devices. 
These input devices are optional since digitized video and audio samples 
may be input into the system by keyboard 15 or disc drive 13 or may be 
resident in ROM 11. 
Referring now also to FIG. 2, a conceptual block diagram of the animated 
synthesized actor, hereinafter referred to as synactor, editing or 
authoring and application system according to the principles of the 
present invention is shown. The animation system of the present invention, 
hereinafter referred to as "hyperanimator", is a general purpose system 
which provides a user with the capability to create and/or edit synactors 
and corresponding speech scripts and to display on a frame-by-frame basis 
the synactors thus created. The hyperanimation system provides animation 
and sound synchronization automatically and in real time. To accomplish 
this, the hyperanimator system interfaces with a real time random access 
driver (hereinafter referred to as "RAVE") together with a descriptive 
authoring language called "RAVEL" which is implemented by the system shown 
in FIG. 1. 
Prototype models, up to eight different models, for synactors are input via 
various input devices 31. The prototype models may comprise raw video 
and/or audio data which is converted to digital data in video and audio 
digitizers 33 and 35 or any other program data which is compiled by a 
RAVEL compiler 37. The prototype synactors are saved in individual 
synactor files identified by the name of the corresponding synactor. The 
synactor files are stored in memory 39 for access by the hyperanimator 
system as required. Memory 39 may be a disk storage or other suitable 
peripheral storage device. 
To create a new synactor or to edit an existing prototype synactor, the 
hyperanimator system is configured as shown by the blocks included in the 
CREATE BOX 30. The author system shell 41 allows the user to access a 
prototype synactor file via RAM 20 and display the synactor on a number of 
screens which will be described in detail hereinbelow. Utilizing the 
various tools provided by the screens and the script command controller 
43, the user is able to create a specific synactor and/or create and test 
speech and behavior scripts to use in an application. The new synactor 
thus created may be saved in the original prototype file or in a new file 
identified by a name for the new synactor. The synactor is saved as a part 
of a file called a resource. Scripting created, for example, digitized 
sound "recite" commands can be saved to application source files by means 
of "clipboard" type copy and paste utilities. The microprocessor 10 
provides coordination of the processes and control of the I/0 functions 
for the system. 
When using a synactor, as an interactive agent between a user and an 
applications program, for example, the hyperanimator system is configured 
as shown by the USE BOX 40. User input to the applications controller 45 
will call the desired synactor resource from a file in memory 39 via RAM 
20. The script command controller 43 interprets script from the 
application controller 45 and provides the appropriate instructions to the 
display and the microprocessor 10 to use. Similarly, as during the create 
(and test) process, the microprocessor 10 provides control and 
coordination of the processes and I/0 functions for the hyperanimator 
system. 
Referring now to FIG. 3, a functional block diagram illustrating the major 
data flows, processes and events required to provide speech and the 
associated synchronized visual animation is shown. A detailed description 
of the processes and events that take place in the RAVE system is given in 
co-pending U.S. patent application Ser. No. 06/935,298 which is 
incorporated by reference as if fully set forth herein and will not be 
repeated. The hyperanimator system comprises the author system shell 41, 
the application controller 45, the script command processor 49 and 
associated user input devices 47 and is interfaced with the RAVE system at 
the script command processor 49. In response to a user input, the 
application controller 45 or the author system shell 41 calls on the 
microprocessor 10 to fetch from a file in memory 39 a synactor resource 
containing the audio and visual characteristics of a particular synactor. 
As required by user input, the microprocessor will initiate the RAVE sound 
and animation processes. Although both the author system shell 41 and the 
application controller 45 both access the script command processor 49, the 
normal mode of operation would be for a user to utilize the author system 
shell 41 to create/edit a synactor and at a subsequent time utilize the 
application controller 45 to call up a synactor for use (i.e., speech and 
visual display) either alone or coordinated with a particular application. 
The hyperanimator system is a "front end" program that interfaces the 
system shown in FIG. 1 to the RAVE system to enable a user to create and 
edit synactors. The system comprises a number of screen images (sometimes 
referred to as "cards") which have activatable areas referred to as 
buttons that respond to user actions to initiate preprogrammed actions or 
call up other subroutines. The buttons may be actuated by clicking a mouse 
on them or other suitable methods, using a touch-screen for example. The 
screen images also may have editable text areas, referred to as "fields". 
The hyperanimator system comprises a number of screens or cards which the 
user moves between by activating or "pressing" buttons to create, edit and 
work with synactors. 
Referring now to FIGS. 4, 5a-5i and 6a-6f, FIG. 4 is a functional block 
diagram illustrating a hierarchical overview of the hyperanimator screens. 
The startup screen 51 comprises one card and informs a user that he or she 
is running the hyperanimator system. The startup screen also provides the 
user with bibliographic information and instructions to begin use of the 
hyperanimator system. Once the initiate button (not shown) has been 
pressed, the RAVE driver is called to perform system checks. The RAVE 
driver is a portion of the hyperanimator system that handles much of the 
programmatic functions and processes of the synactor handling. It 
introduces itself with a box message (not shown) which includes a "puppet" 
icon. After the initial checks have been passed, a star screen 53 is shown 
which provides a transition between the startup screen 51 and the menu 
screen 55. The menu screen 55 is then shown after the star screen 53. The 
startup screen 51 also includes a button (not shown) for taking the user 
to the hyperanimator credit screen 57. The credit screen 57 comprises one 
card and provides additional bibliographic information to the user. The 
credit screen 57 can be accessed three ways: from the startup screen 51, 
from the menu screen 55 and from the first card in the dressing room 59. 
Pressing or clicking anywhere on the credit screen 57 will take the user 
back to the card he or she was at before going to the credit screen 57. 
The menu screen 55 (also shown in FIG. 5a) comprises one card and is 
provided to allow the user to navigate among the hyperanimator system 
features. Upon first entering the menu screen 55, the Hyperanimator 
Navigator 510 greets the user. The menu screen 55 contains seven buttons 
for accessing the hyperanimator system. 
The seven buttons allow the user to: go to the dressing room 59, go to the 
casting call screen 67, go to the sound booth screen 63, go to the speech 
sync screen 65, go to the credit screen 57, and quit 513 the hyperanimator 
system. With the exception of the quit button, the buttons take the user 
to different cards within the hyperanimator system. The quit button closes 
hyperanimator and returns the user to the host operating system shell 
level in the host program. Anytime the user returns to the menu screen 55 
from within the hyperanimator system, the HyperAnimator Navigator 510 will 
greet him or her. 
The casting call screen 61 (also shown in FIG. 5b) comprises functions 
which allow the synactor files to be copied or deleted from memory 39 or 
placed in the dressing room 59. An appropriate designed button 521, 523 
and 535 represents and initiates each of these tasks. Copying a synactor 
file takes the file resource of a selected synactor from an application 
program or synactor file and places an exact copy in a destination 
application program or synactor file. (A synactor file is defined as a 
file containing synactor resources only.) Placing a synactor into the 
dressing room 59 (also shown in FIG. 5c) allows the user to edit an 
existing synactor. The user selects a synactor from an application program 
or synactor file stored in memory 39. Deleting a synactor removes a 
selected synactor resource from an application program or synactor file in 
memory 39. The RAVE driver includes special commands to accomplish the 
tasks initiated at the casting call screen 61. 
The sound booth screen 63 (also shown in FIG. 5f) comprises functions which 
allow sound resources to be copied or deleted from a file. Sound resources 
are portions of files which are sequential prerecorded digital 
representations of actual sound. They are input to the system via digital 
recording devices and stored as resource files in memory 39. An 
appropriately identified button 527, 529 initiates these functions. The 
sound booth screen also provides buttons 531, 533 to allow the user to 
return to the menu screen 55 and the speech sync screen 65. 
The dressing room screen or dressing room 59 begins with an animated 
sequence (not shown) showing a door opening into a room. The dressing room 
59 is used to create new synactors or to edit existing synactors. A user 
can access the dressing room 59 from the menu screen 55, from any Face 
Clip Art card 75, from the stage screen 77, from the spotlight screen 79 
or from the casting call screen 61. The dressing room proper comprises 
sixteen cards 71. Placing a synactor into the dressing room 59 places each 
image 83 of the selected synactor in the synactor easel 85 on the 
respective cards 71 in the dressing room 59. For example, the REST image 
83 is placed on the REST card 87 and the REST button 89 is highlighted. 
Each synactor will have sixteen images corresponding to respective ones of 
the sixteen cards 71 of the dressing room 59. Each of the sixteen cards 71 
contains two buttons allowing the user to return to the menu screen 55 and 
go to the stage screen 77. Each of the sixteen cards 71 also includes a 
button 95 for taking the user to the Face Clip Art menu screen 73. Each of 
the sixteen cards 71 contain a field 97 at the top informing the user that 
he or she is currently in the dressing room 59. Each of the sixteen cards 
71 includes a representation of a painter's easel called the synactor 
easel 85. Each of the sixteen cards 71 includes sixteen buttons 72 which 
represent each of the sixteen cards 71. 
With these buttons 72, the user can immediately go to any of the sixteen 
cards 71 from any of the sixteen cards 71 within the dressing rom 59. For 
each of the sixteen cards 71, the button that represents itself is 
highlighted showing the user where they are within the dressing room 59. 
Each of the sixteen cards 71 has a field 99 which labels which of the 
sixteen cards it is. The sixteen cards 71 which make up the dressing room 
59 are labeled as follows: REST, F, M, R, W, IH, AH, E, Al, A2, A3, A4, 
A5, A6, A7, AND A8. 
The first eight cards deal with specific lip positions which correspond to 
the sounds of the letters that the cards represent. The last eight cards 
deal with any type of expression. The first eight cards each contains a 
picture in the field 99 of representative lips which indicate the lip 
position corresponding to the letter that card represents. The last eight 
cards contain the saying "Expressions" because expressions are not 
predefined (the user can design the expressions as desired; smiles or 
frowns, for example). The REST card 87 also has a special button 101 which 
enables the user to copy the image 83 that resides on the synactor easel 
85 within the REST card 87 to the synactor easel 85 on every card within 
the dressing room 59. This button 101 is only present on the REST card 87. 
Each of the sixteen cards 71 in the dressing room 59 also include a menu 
103 which allows access to additional tools such as paint tool or 
scrapbook applications which the user can manipulate to create or edit 
synactors. Pressing the stage button 93 on any of the dressing room's 
sixteen cards will initiate the building and copying of the synactor in 
the dressing room 59 into a temporary memory (not shown) and take the user 
to the stage screen 77 to display that synactor. No matter where the user 
is located within the dressing room 59, pressing the stage button 93 
always selects the REST card 87 to begin building and copying the synactor 
into memory. When building the synactor, the art that is within the frame 
of the synactor easel 85 on the REST card 87 is selected and copied first. 
The hyperanimator system then calls on an external command (XCMD) which 
provides the memory location where that image is stored. The next dressing 
room card is then selected and the above procedure is repeated. Each of 
the sixteen dressing room cards is selected in sequence and the art within 
the frame of the synactor easel is copied. When all of images have been 
copied, a list of the memory locations for the images is sent to the RAVE 
driver where a synactor resource is built of those images in memory. At 
the completion of the synactor resource file building process, the user is 
transferred to the stage screen 77 to view the synactor thus created. 
The stage screen 77, 78, 81 is a display for examining the 
lip-synchronization of newly constructed synactors. It is entered by 
pressing the appropriate button 93 found on any of the sixteen cards 71 of 
the dressing room 59. The stage screen consists of eight cards 77, 78, 81 
of which seven are used for animation purposes (not shown). The first five 
cards 77 show stage curtains opening up. The sixth card 78 (also shown in 
FIG. 5d) is an open stage 105 where a newly created synactor 107 is 
displayed. 
The stage screen 78 provides a button 109 and a field 111 which allow the 
user to enter in any text string and see and hear the synactor 107 speak. 
The "Read Script" button 109 takes the text string entered in the field 11 
and calls the RAVE driver to create the animation and speak the text 
string through the RAVE system. The stage screen 78 contains three buttons 
113, 115, 117 allowing the user to return to the menu screen, return to 
the dressing room, or go on to the spotlight screen 79, respectively, to 
save the newly constructed synactor 107. 
If the user chooses to return to the menu screen, the newly constructed 
synactor is retired and the HyperAnimator Navigator 510 is returned. If 
the user chooses to return to the dressing room 59, the two remaining 
cards 81 in the stage screen are called showing the synactor being pulled 
from the stage 105 with a stage hook. If the user would like to save the 
synactor to a destination program or synactor file, the user should click 
or press on the spotlight screen button 117. 
The spotlight screen 79 consists of one card (also shown in FIG. 5e) and 
allows the user to save a newly constructed synactor as a resource file. A 
newly constructed synactor exists as temporary data in RAM memory and must 
be saved permanently to a file or be lost. The spotlight screen 79 
provides a field 119 where the user can type in a text string that will be 
the new synactor's file name. The text string must be one continuous word. 
The spotlight screen 79 has a "Save Actor" button 121 that allows the user 
to select a destination program or synactor file to save the newly 
constructed synactor resource in. If the destination program or synactor 
file already contains a synactor with the same name as the text file in 
the spotlight screen field 121, a different name must be selected or the 
existing synactor file will be lost. After the newly constructed synactor 
is saved, the user is taken back to the menu screen 55. The spotlight 
screen 79 also includes two buttons 123, 125 which allow the user to 
return to the menu screen 55 or to return to the dressing room 59. 
Art which can be used to create synactors is provided within the 
hyperanimator system in a Face Clip Art screen 73, 75. The Face Clip Art 
screen comprises seventeen cards; one, shown in FIG. 6a, serves as a menu 
for navigating among the Face Clip Art cards 75 and the other sixteen 
cards 75 contain the actual art, examples of which are shown in FIGS. 6b 
and 6c. The Face Clip Art screen can be entered from any of the dressing 
room cards 71 through a Face Clip Art button 95. Upon entering the Face 
Clip Art screen, the user is first taken to the Face Clip Art Menu 73. 
From the Face Clip Art Menu 73, the user can directly access any of the 
sixteen cards 75 containing Face Clip Art. The user can also return to the 
dressing room 59 from the Face Clip Art Menu 73. Each of the sixteen Face 
Clip Art Cards 75 behaves in a similar manner. Each Face Clip Art Card 75 
has a button to return to the dressing room 59. Each Face Clip Art Card 
has a button to search linearly left through the Face Clip Art Cards and a 
button to search linearly right through the Face Clip Art Cards. Each Face 
Clip Art Card also has a button to return to the Face Clip Art Menu card. 
Each Face Clip Art Card has a title field 137 identifying which of the 
sixteen types of Face Clip Art it contains. 
Each Face Clip Art Card also provides a utility for automatically copying 
any piece of art into the card of the dressing room where the user was 
last at before entering the Face Clip Art Screen. The user selects a piece 
of art by clicking on it with a mouse. The hyperanimator system then takes 
the user to the dressing room and asks the user to indicate where the art 
should be placed. The user can then drag the art around within the 
dressing room card to fine tune its placement. The user then clicks where 
the art should be placed and the selected Face Clip Art appears. 
The sixteen types of Face Clip Art found within the Face Clip Art Section 
are as follows: the eight lip positions: REST, F, M, R, W, IH, AH, E, Eye 
Clip Art, Blink Clip Art, Nose Clip Art, Eyebrow Clip Art, Ear Clip Art, 
Miscellaneous 1 Clip Art, Miscellaneous 2 Clip Art, and Miscellaneous 3 
Clip Art. Clicking on an art image while on the REST Clip Art Card allows 
the user to copy just that REST image, or copy all of the lip positions 
associated with that mouth. 
The tutorial screen 67 consists of one card (shown in FIG. 5h) and is used 
to introduce the basic RAVE language commands to the user. The tutorial 
screen 67 is accessed by a button on the menu screen 55. The tutorial 
screen 67 includes four arrows which are clicked on by the user to 
introduce four RAVE commands. The HyperAnimator Navigator 510 briefly 
describes each of these four commands. Additional information about each 
of the four RAVE commands is provided in a field on the tutorial screen. 
The tutorial screen contains a button for returning back to the menu 
screen. 
Synthesized speech, as used in the stage cards, is automatically 
synchronized by RAVE. For digitized sounds, the process of ensuring that 
the face has the correct lip position at the time the sound is being made 
is called speech synchronization and is performed in the "speech sync 
laboratory" represented by the speech sync screen 65. The speech sync 
screen 65 (shown in FIG. 5g) comprises one card and enables the user to 
create RAVE RECITE commands. The speech sync screen 65 can be accessed 
from the menu screen 59 and from the sound booth screen 63. The speech 
sync screen contains three fields and three buttons. The speech sync 
screen also contains two buttons allowing the user to return to the menu 
screen 55 or go to the sound booth screen 63. 
The user enters a text string that represents the sound he or she is 
synchronizing in a first field 535, Text String. A button 537, CONVERT 1, 
is provided to translate this text string 536 into a phonetic text string 
538. The phonetic text string 538 is placed into a Phonetic String field 
539. The phonetic text 538 can be modified or edited by the user within 
the second field 539. A second button 541, CONVERT 2, is provided which 
allows the user to select which sound resource and file he or she is 
using. The hyperanimator system then sends the phonetic string along with 
the location of the sound resource and file to the RAVE driver. The RAVE 
driver uses this information to automatically create an approximate RAVE 
RECITE command 542 which is displayed in the Talk Command field 543. The 
RAVE RECITE command 542 contains all the information that is needed to 
place the command within the script and have it run properly. The RAVE 
RECITE command contains the key words, "RAVE" and "RECITE". It also 
includes the name 544 of the sound resource that is played when the 
command is issued. The final element of the RAVE RECITE command 542 is the 
phonetic/timing value string 546. 
The phonetic/timing value string 546 contains various phonetic/timing value 
pairs. The first element of a phonetic/timing value pair is a phonetic 
code (one or two letters). The second element of a phonetic/timing value 
pair is an integer number. The phonetic code tells the RAVE driver which 
face or faces to display according to the sequences and other tables in 
its precompiled synactor model. The integer number tells the driver how 
long to display that face on the screen with units comprising time ticks. 
A tick has a value of approximately 1/60th of a second. Therefore, a 
timing value of 30 lasts half a second. The RAVE RECITE command 542 is 
sent from the RAVE driver to the hyperanimator system. The hyperanimator 
system then puts the RAVE RECITE command 542 into the final field 543 on 
the speech sync screen, the Talk Command field. The speech sync screen's 
last button 545, TEST 3, allows the HyperAnimator Navigator 547 to use the 
RAVE RECITE command found in the Talk Command Field. 
The speech sync screen provides the user with three ways to modify the RAVE 
RECITE command. The user can select any phonetic letter or timing number 
and delete or replace it with characters/numbers entered from a keyboard 
15 (as shown in FIG. 1). If the user changes any timing value, depressing 
the return key directs the RAVE driver to recalculate the timing values 
for the entire phonetic/timing value string after the change. Entering a 
stop character (.) anywhere in the Recite String instructs the RAVE driver 
to recalculate the timing values for the phonetic/timing value string only 
up to the stop character. The speech sync screen also allows the user to 
select any portion of the phonetic/timing value string and hear and see 
the corresponding portion of the digitized sound and synchronized 
animation. The user selects at least one entire phonetic/timing value pair 
and presses the final speech sync screen button 545. This directs the RAVE 
driver to play the sound and animation for the duration of the selection 
only. If text is selected from the Text String field 535 and the return 
key is depressed, the hyperanimator system will select and highlight the 
corresponding text below in the Phonetic String field 539. Also if text is 
selected from either Text String field 535 or the Phonetic String field 
539 and the return key is depressed, the hyperanimator system will select 
and highlight the corresponding phonetic/timing value pairs below in the 
Recite String field 543. Upon leaving the speech sync screen, the 
hyperanimator system prompts the user to prevent accidental loss of data. 
The speech sync screen 65 may be enhanced by including additional 
capabilities to digitizing sounds such as a sound waveform display window 
and subroutines for editing or tailoring the waveform analogous to the 
tuning of the phonetic/timing value string described above. 
The steps involved in creating a synactor with hyperanimator are summarized 
as follows. Synactors are created in the hyperanimator system dressing 
room. Assuming that a user is at the menu screen 55, clicking on the 
dressing room button will take the user to the REST image card, the first 
card in the dressing room. The dressing room contains all the tools 
necessary to create a synactor. 
First, the REST image of the synactor is created. After the REST image is 
created, it can be copied and used as a template for the other fifteen 
images. A paint tool utility may be used to draw the outline of the 
synactor within the synactor easel. The paint tool utility is 
automatically presented when the user first enters the dressing room. Any 
of the paint tools can be used to create the synactor. The hyperanimator 
system also provides Face Clip Art which can be used in creating the 
synactors. Clicking on the Face Clip Art button on any of the dressing 
room cards transfers the user to the Face Clip Art menu screen. The Face 
Clip Art menu screen contains sixteen buttons which represent the sixteen 
cards of Face Clip Art which make up the Face Clip Art screen. Clicking on 
the Eye Images button transfers the user to the Eye Images card. Clicking 
on any one of the images provided on this card selects that image and 
transfers the user back to the dressing room. The hyperanimator system 
then prompts the user for placement of a copy of the selected image. 
Clicking anywhere on the dressing room card and the selected image will 
appear. In a similar manner, art images can also be copied into the 
dressing room utilizing a standard scrapbook facility. 
When the REST image of the synactor is complete, click on the Copy REST to 
All button. This button will place a copy of the REST image on each of the 
fifteen remaining dressing room cards. The Copy REST to All button is only 
found on the first dressing room card. By using the REST image as a 
template for the other cards, the amount of work required to create a 
synactor is reduced. 
Next, return to the Clip Art Menu screen. Click on the REST Images button. 
The REST Images card provides a collection of mouths which can be added to 
the synactor. Select one of the mouths. The hyperanimator system will the 
user to copy all of the mouth positions. Select All. The hyperanimator 
system then transfers the user back to the dressing room and copies mouth 
images on each of the sixteen cards within the dressing room. The correct 
lip position is matched with and copied on the correct card automatically. 
The sixteen buttons located on each card in the dressing room allow the 
user to go to any dressing room card. Clicking on the A1 button highlights 
the A1 button and the user is transferred to the A1 expression card. Each 
expression card is individually created for a desired expression. For 
example, the user can utilize the paint tool to change the image on the 
Synactor Easel to make the synactor look as if it is sleeping. 
The synactor will be finished when the user completes all of the expression 
images. The synactor then can be built and placed into a temporary memory 
file by clicking on the stage button. The stage button is located in the 
lower right corner of any dressing room card and resembles the stage card. 
In a second preferred embodiment of the present invention, the capability 
and versatility of the dressing room 59 and the speech sync screen 65 have 
been greatly expanded. Utilizing a relatively small amount of memory, 
synactor models composed of 16 images (cards), 8 devoted to speaking and 8 
devoted to animated expressions can be created. Relaxing the memory 
restrictions, greatly enhanced synactors, including color synactors, 
having the following selectable image configurations (speaking/expression 
image mix) may be created. 
TABLE 1 
1) 8 lip positions/16 total images 
2) 16 lip positions/16 total images 
3) 8 lip positions/32 total images 
4) 16 lip positions/32 total images 
5) 8 lip positions/64 total images 
6) 16 lip positions/64 total images 
7) 8 lip positions/127 total images 
8) 16 lip positions/127 total images 
Referring now to FIGS. 7a and 7b, FIG. 7a is a presentation of the REST 
card 710 in the dressing room 59 having enhanced capabilities. Since there 
may be as many as 127 total images of a synactor, there will be a separate 
card corresponding to each of those images totalling as many as 127 cards. 
The configuration of a particular synactor, i.e., the number of images and 
the mix of the images between lipsynch positions and other facial 
expressions is determined by the setting of the four buttons 711 located 
on the REST card 710. Pressing any of the four buttons 711 will cause the 
number of cards in the dressing room to increase, decrease or remain the 
same depending on its present state. After a particular button 711 is 
pressed, the number of lipsynch positions, 8 or 16, has to be selected. 
The key label 712 and image label 713 that describes each card in the 
dressing room will change as a function of the number of lipsynch 
positions and total images selected. Each of the combinations shown in 
Table 1 above represent the selection of a different precompiled prototype 
synactor model which has been stored in memory 39. Four buttons 705 are 
provided to allow the user to shift or navigate between the cards. Image 
number field 706 indicates which card the user is presently located on. 
Moving between the image cards is accomplished in four ways. Clicking on a 
single right/left arrow takes the user to the nest/previous card. Clicking 
and holding on a single right/left arrow flips through the next/previous 
cards one-by-one. Clicking and holding on double right/left arrows quickly 
jumps through the next/previous cards. To jump to a card directly, click 
on the image field 706, enter a valid image card number and hit return. 
When a synactor is placed in the dressing room 59 from the casting call 
screen 61, the dressing room 59 will automatically change its 
configuration to match the configuration of the selected synactor. The 
RAVEL code for two of these possibilities is shown in Appendix I and 
Appendix II. Appendix I is the code for a synactor model having 8 lipsynch 
positions and 8 other expressions for a total of 16 images. Appendix II is 
the code for a synactor model having 16 lipsynch positions and 111 other 
positions for a total of 127 images. 
The physical size of a synactor appearing in the dressing room can be 
changed. The height of a synactor can be any value while the width is 
limited to 32-pixel boundaries. The synactor palette 715 can be altered in 
three ways: click on either of the height/width values, 714, 716, 
respectively, changing the number to the desired value and hit return; 
click and drag within the synactor palette 715 to move it; click and drag 
within the lower right hand corner of the synactor palette to change the 
size of the synactor palette 715. While moving the synactor palette 715, 
the synactor palette move button 708 will be highlighted to indicate that 
the mode is active. Clicking in the lower right hand corner of the 
synactor palette 715 will allow the user to change the size of the 
synactor palette. While changing the synactor palette 715, the synactor 
palette zoom box button 709 will be highlighted to indicate that the mode 
is active. 
The elements, Synactor Palette Height Value 714, Synactor Palette Width 
Value 716, Synactor Palette Zoom Box 709, Synactor Palette Move 708, 
Synactor Palette Undo 707, Total Image Number 711, and Copy REST to All 
718 are all unique to the REST card 710 of the dressing room. Certain 
actions have a global effect on the dressing room. For example, changing 
the height of the synactor palette 715 changes the height of the synactor 
palette for all cards within the dressing room. Because of their global 
nature, it is important that the above seven elements above be located in 
a specific and easily accessible location. 
Dressing rooms having more extended or exotic features to cover special 
cases may be utilized. For example, a dressing room could provide enhanced 
facilities to allow a user to conveniently work with very large or 
coarticulated synactors. Referring now also to FIG. 7b, for example, a 
special case dressing room screen 750, referred to as a portrait studio 
750, may be utilized for easily handling of synactors comprising digitized 
video images. The portrait studio screen 750 includes a synactor easel 751 
having a current image 753 displayed thereon. A key image 755 which 
represents the lip position to be scanned along with a phonetic label 757 
of that lip position are provided to identify each card (image) in the 
portrait studio 750 corresponding to the selected synactor 753. Navigation 
buttons 759 operated in the manner described hereinabove are provided to 
allow a user to maneuver through the images which make up the synactor; a 
numeral 761 indicates which card (image) the user is presently in. The 
number of images involved with a synactor 753 can be varied in accordance 
with Table 1 above from 16 to 127 utilizing buttons 763. The REST image 
(not shown) can be copied to all of the other images, i.e., cards, with 
the COPY REST to ALL button 765. The SCAN IMAGE button 767 scans 
(digitizes) a desired image and places the image 753 on the current card 
750. The BUILD SYNACTOR button 769 creates a synactor from the scanned 
images. The VIEW SCAN button 771 allows the user to examine and adjust the 
current scan. The SAVE ACTOR button 773 allows the current synactor to be 
saved to memory without going through the stage screens 77, 78. The MALE 
VOICE button 775 allows the current synactor to speak with a previously 
digitized male voice. The FEMALE VOICE button 777 allows the current 
synactor to speak with a previously digitized female voice. 
Referring now to FIG. 8, a synactor model table 810 is illustrated. A 
number of internal data structures are compiled from the RAVEL source 
program and stored in RAM 20 to provide the RAVE with sufficient data to 
implement its functions. A dynamically allocated synactor model table is 
the basis for these data structures and contains one ore more synactor 
model table records 810, one for each synactor model which has been 
defined. The synactor model table 810 defined for each synactor is 
included in that synactor's file stored in memory 39. 
The first field in each synactor model table record 810 is the Owner field 
801. This field contains the synactor model number (as defined in the 
RAVEL source), or a special code to indicate that this entry is empty (for 
table management). The next field, Reader 803, points (i.e., specifies the 
address where the reader table is located) to the reader table for that 
model. Reader tables consist of a header in which special codes are 
defined, followed by the rules, stored in compressed concatenated fashion. 
The next field, phocodes 805, specifies a pointer to the photable for this 
model. This photable is a lookup table which defines the narrator device 
characteristics of the synactor model in terms of its speech segment and 
other codes. Each code has its own record in the photable plus a filler 
record for phocode zero which is not assigned. The first field in that 
record specifies the number of bits in that particular narrator device 
code. The next field lists the bits that are used to define that code to 
the narrator device, and finally there is a zero terminator. The next 
entry in the synactor model table, Phocodes Count 807, is the number of 
records in the photable. 
The next field in the synactor model table, Syncopations 809, points to the 
syncopations table, which describes the sounds necessary to sound out a 
word. This is a count off table containing one syncopation for each 
phocode plus a filler record for phocode zero. This count off table is 
delimited with a dedicated delimiter code after each syncopation string. 
The next field in the synactor model table, Syncopations Count 811, 
specifies how many bytes the syncopation table takes up. The Syncopations 
Count 811 is required for the synactor model table management routines in 
order to reallocate the table when a model is discarded. The next field in 
the synactor model table, Sequences 813, points to the sequences table. 
This is a count off table, counted off by phocode, having a filler entry 
for phocode zero, separated by a dedicated code, each entry of which 
specifies the positions and timing values as given for that phocode in the 
RAVEL source file. Each of the entries in the Sequences table consists of 
zero or more pairs of values. A zero-length "empty" entry in a count off 
table is denoted by concatenated separator codes. The first value in each 
pair is a position number that will index the positions table to find a 
pointer to a screen image or other synactor animation block (SABLOCK) data 
for that position; the second value is the time for which it is to be 
displayed on the screen (it may be displayed for a longer period if 
necessary to wait for an event). The next field in the synactor model 
table, Sequences Count 815 specifies the number of bits in the sequences 
table. 
The next field in the synactor model table, Betweens 817, is a pointer to 
the inbetweens table. This is a linear list of records called inbetween 
records. Inbetweens are intermediate images which are displayed to smooth 
the transition between two images. Each has four associated values: The 
first position number, the second position number; the position number 
that is to be inserted between those two positions; the Time field for 
that inbetween, or zero to indicate a default to half the previous time in 
a synactor animation script (or to zero if the previous time is one cycle 
or less). This table is not indexed, it is sequentially scanned only. The 
next field in the synactor model table, Betweens Count 819, specifies the 
number of records in the inbetweens table. The next two fields, Width 821 
and Height 823, specify how large the synactor image is to be on the 
screen. The next field, Model Flags 825, is used to indicate specialized 
modes of synactor animation. 
The next field, Position Pointers 827, points to the positions table for 
this synactor model, which is indexed by the position numbers form the 
other tables to yield pointers to SABLOCKs. The first entry in the 
positions table is not used. Position number zero is illegal. The next 
entry, position number one, is dedicated for the initial "at rest" 
position. Additional position numbers are assigned arbitrarily by the 
programmer. 
SABLOCK data is animation means dependent data. In the preferred 
embodiment, the SABLOCK contains the data required to generate the screen 
image of a particular position of the synactor, for example, bitmaps, 
encoded display values or other parameters for image synthesis. For other 
embodiments, such as robotic means, the SABLOCK may contain commands to 
actuators or the like that would move various parts of the robot. 
The next value in the synactor model table, PhoFlags 829, points to a table 
of phocode attribute records indexed by phocode. 
The next field in the synactor model table, Characteristics 831, is a 
pointer to a block specifying the narrator device number for this synactor 
model, as given in the RAVEL source code, and narrator device dependent 
data. This would normally contain values for speed, pitch, volume and 
various other attributes peculiar to a particular narrator device in a 
format convenient to the audio processor. 
The next field Coarticulations 833 is a pointer to a look up table used for 
determining the different face (position) to be used if a consonant is 
followed by two different vowels. The lookup value is found using a 
combination of the phocode index and the vowel that is affecting the 
phocode. A detailed description of the coarticulation process is given 
hereinbelow. Coarticulations Count 835 is the number of faces in the above 
list. Coart Types 837 is a pointer to a lookup table used for determining 
what effect the given phocode will have on its neighbor. 
Face Top Left 839, designates a Point representing the top left coordinates 
of where the synactor was created in the dressing room. This field is 
changed dynamically while the synactor is being shown, and will represent 
the top left coordinate of the synactor after being moved around the 
screen. These dynamic changes do not affect the permanently stored value 
of the synactor's dressing room position. The Depth 841 refers to the 
number of bits per pixel of information that this synactor was created 
with. Computer monitors have a number of colors dependent on how many bits 
are used to represent each individual pixel of information. The more bits, 
the more varieties of color. Face Window 843 is a pointer to the host 
operating system window structure that is used to mark where the synactor 
should be drawn. Model Name Handle 845 is a pointer to a pointer to a 
string holding the name of this synactor and is used with RAVE commands 
that specify the name of the synactor that a particular command or 
particular set of commands is to be performed on. 
The last two fields refer to the dynamic movement of the synactor from a 
resource block in memory 39 (such as on a disk) to being active in RAM 20. 
The Model Block Pointer 847, is a pointer to this block of information 
which can be referenced to determine if this synactor has just been read 
into memory or not. Zero Bias 849: When a synactor is saved as a resource, 
or a block of information to memory 39, the pointer fields are all set up 
as offsets to the actual information in the block. When the block is read 
back into RAM 20, these offsets are all resolved into pointers again, 
adding the offset values to the model block pointer 851 value in the field 
described above. This field indicates whether or not this relocation has 
taken place yet. 
Referring now to FIGS. 9 and 10a-10d, FIG. 9 is a functional block diagram 
of the processes in the speech sync screen or lab 65 (as shown in FIG. 4). 
FIGS. 10a-10d are 4 presentations of different speech sync screens 
illustrating the processes of selecting the text for a synactor to speak, 
converting to phonetics, selecting the digitized sound to synchronize, 
adjusting or tuning the timing values and testing the synchronization 
between the synactor animation and its speech. 
The user can choose a synactor to be used for testing when in the speech 
sync lab. Clicking on the HyperAnimator Navigator provides a dialogue box 
which prompts the user to type in the name of the new synactor. The named 
synactor will replace the regular speech sync synactor 1023 (the 
HyperAnimator Navigator) on the screen. This allows the user to 
synchronize sounds with any specific synactor or with synactors having 
different attributes. For example, the HyperAnimator Navigator is an 8/16 
synactor. The user may have created a 16/x synactor and want to 
synchronize speech with its more complicated faces. 
The first step in synchronizing a digitized sound resource from a file is 
to enter a text string 1012 which represents the sound recorded in the 
sound file. The text string 1012 is entered in the first field 1011 titled 
"Text String". Type "Good Afternoon" into the Text String field 1011. 
Clicking on the CONVERT 1 button 1017 will allow the hyperanimator system 
to convert a text string into its phonetic translation. The text string 
1012 in the Text String field 1011 is converted into a phonetic string 
1014 and placed into the Phonetic String field 1013. The phonetic string 
1014 looks like this: "GUH5D AEFTERNUW5N". Clicking on the CONVERT 2 
button 1019 will allow the hyperanimator system to convert the phonetic 
string 1014 into a RAVE RECITE string 1016. The hyperanimator system first 
prompts the user with a directory listing 1025 (as shown in FIG. 10b) to 
identify the location of the sound resource and file that will be 
synchronized. The sound file "GoodAfternoon" is located in the 
hyperanimator program. Select hyperanimator and click on to open. The 
hyperanimator system then prompts the user with a directory listing 1027 
(as shown in FIG. 10c) to identify which sound file is to be used. Select 
the"GoodAfternoon" sound file 1026 and click on "Select". The phonetic 
string 1014 in the Phonetic String field 1013 is converted into a RAVE 
RECITE string 1016 and placed into the Talk Command field 1015. The RECITE 
string looks like this: RAVE "{RECITE GoodAfternoon G 2 UH 5 D 5 AE 6 F 5 
T 4 ER 7 N 5 UW 5 N 4}". Clicking on the Test 3 button 1021 directs the 
hyperanimator system to execute the RECITE string 1016. The RECITE string 
1016 makes the HyperAnimator Navigator 1023 pronounce "Good Afternoon" 
using the GoodAfternoon sound file 1026 that was selected. 
The Recite command string 1018, which appears in the Command Field 1015, 
consists of a series of phonetic/timing pairs. The timing values attached 
to each phonetic element are what determine the quality of the animation 
and synchronization. To allow correction of these values, the 
hyperanimator system provides three features: one for testing and two for 
the actually tuning, or adjustment of the timing values. By selecting any 
portion of the Recite command string 1018, in phonetic/timing pairs, and 
clicking on the TEST 3 button 1021, that portion of the sound will be 
pronounced. Using the Speech Sync process, the GoodAfternoon sound can be 
converted to the following Recite Command 1016: G 2 UH 5 D 5 AE 6 F 5 T 4 
ER 7 N 5 UW 5 N 4. If "G 2 UH 5 D 5" 1022 is selected, the animation for 
"good" will be performed and the corresponding portion of the sound 
pronounced (i.e., the first 12 ticks of the entire Recite command). If " N 
5 UW 5 N 4" 1024 is selected, "noon" will be pronounced. Similarly, if "ER 
7" 1026 is selected, "er" will be pronounced. Two features which exist in 
the speech sync lab for convenient tuning beyond simple editing aid in 
maintaining the total time value of the strings. When a timing value has 
been changed and the return key depressed, the amount that the timing 
value changed will be adjusted throughout the phonetic/timing pairs from 
the cursor to the end of the Recite command. Additionally, by placing a 
bullet, "." 1027 between two phonetic/timing pairs, the timing adjustment 
will be spread between the cursor 1025 and the bullet 1027. Using the same 
Speech Sync example as above, G 2 UH 5 D 5 AE 6 F 5 T 4 ER 7 N 5 UW 5 N 4, 
the timing value for the "T" 1029 will be adjusted. First the cursor 1025 
is positioned after and adjacent the timing value 1031 for the "T" 1029 
and, via the keyboard 15, the initial timing value for the "T" 1029 is 
changed from "5" to "8" 1031 and the return key depressed, leaving the 
cursor in position at the "T". The result will be: G 2 UH 5 D 5 AE 6 F 5 T 
8 ER 6 N 4 UW 5 N 3. The timing values after the T have been adjusted to 
make up for the additional time given to the T while maintaining the total 
time for the string at its original value. While the best results are 
typically obtained by working from the beginning of the sound string to 
the end when tuning it, the bullet 1027 feature may be used to hold all 
changes made to the end and go back to adjust the beginning. In the 
example given above, when the portion of the command string from the "T" 
to the end is adjusted, any further changes to the string are blocked by 
inserting the bullet 1027 before the "T" 1029: G 2 UH 5 D 5 AE 6 F 5 . T 
8 ER 6 N 4 UW 5 N 3. Increasing the timing value for the "AE" from 6 to 8 
and depressing the return key results in G 2 UH 5 D 5 AE 8 F 3 . T 8 ER 6 
N 4 UW 5 N 3. The timing values of the string from the cursor 1025 (at the 
"AE") to the bullet 1027 have been adjusted to accommodate the increase of 
the "AE" timing value while maintaining the total string time the same. 
Clicking on the TEST 3 button 1021 at any point during this process, as 
long as there is no selection, will test the whole string. 
The Speech Sync Lab also provides an easy method for viewing text or 
phonemes as they have been converted in the following string fields. If 
text in the Text String field 1011 is selected and return is depressed, 
the corresponding text will be selected in the Phonetic String field 1013. 
Similarly, if a phonetic string is selected in the Phonetic String field 
1013 and return depressed, the corresponding phonetic/timing value pairs 
will be selected in the Talk Command field 1015. This feature allows the 
user to quickly select and highlight word strings and phonetic/timing 
value pairs to isolate portions of the sound and animation for testing. 
Scripts involved in the hyperanimator system Speech Sync Lab are given in 
Appendix III. These scripts handle screen, mouse and keyboard interactions 
and simple logic flow and computation with script code routines called 
handlers. These scripts are activated by specific user actions. Each line 
of script occupies one line. If more than one line is required, the 
option-Return character is included indicating that the script continues 
on the following line. Functions which return data place the data in the 
variable "it". RAVER is a special example of such a function, called an 
XFCN. The RAVER and RAVE subroutines access code in the RAVE runtime 
driver and editing package. The operation of the various RAVE and RAVER 
commands are described in more detail in Appendix IV. 
The CONVERT 1 button sends the text string entered in the Text String field 
to the RAVER XFCN. The RAVER XFCN command "CONVERT" is responsible for 
converting text strings to phonetic strings. RAVER returns the resulting 
phonetic string into a temporary variable. The contents of the variable 
are then placed into the Phonetic String Field. The script in the CONVERT 
1 button first examines the Text String field to make sure it contains 
text. If there is no text, the user is warned and the cursor is placed in 
the Text String Field (1 of Appendix III). The script then builds a RAVER 
CONVERT command by including the text entered in the Text String field. 
The phonetic conversion is placed in a temporary variable and then placed 
in the Phonetic String field (2 of Appendix III). 
The script in the CONVERT 2 button first examines the Phonetic String field 
to make sure it contains text. If there is not text, the user is warned 
and the cursor is placed in the Phonetic String Field. If there is text, 
the script then checks to see if a sound has been selected. If a sound has 
not been selected and processed, then the variable SoundName contains the 
word "empty". If SoundName contains "empty", the script issues a RAVE LOCK 
command which locks down resources in memory. RAVER is called with an 
"OPENSOUNDFILE" command which produces a dialogue box which is used to 
select the digitized sound file used for synchronization. The RAVER 
"OPENSOUNDFILE" command places either the name of a selected sound or 
"false" into soundName. If a sound was properly selected, the name of a 
selected sound is placed in soundName. If a sound was not properly 
selected, "false" is placed in soundName. If a sound was not properly 
selected, the script must stop executing and exit the mouseUp handler. A 
RAVER "SYNCSOUND" command is created with the name of the selected sound 
which is found in the variable soundName. The RAVER "SYNCSOUND" command 
loads the sound into memory. If there is a problem with loading the sound 
into memory, the RAVER "SYNCSOUND" command places "FALSE" into the 
variable testFlag. If a sound could not be loaded into memory, the user is 
warned and the script must stop executing (6 of Appendix III). A RAVER 
"DIGIMAKE" command is then constructed which contains the phonetic string 
located in the Phonetic String field. The RAVER "DIGIMAKE" command returns 
the phonetic/timing value pairs which are a component of a RECITE command. 
These pairs correspond to the phonetic string that was sent to the RAVER 
"DIGIMAKE" command. The hyperanimator system then builds a RAVE RECITE 
command with the returned phonetic/timing values and places it in the Talk 
Script Field. Because the RAVER "DIGIMAKE" returns only phonetic/timing 
value pairs, the string "RAVE "{"is added before the pairs and "}"" is 
added after the pairs. 
The script in the TEST 3 button first examines if a selection of text 
exists in the Talk Command field. If a selection exists, the user wants to 
see a selection of sound. If a selection does not exist, the user wants to 
see the entire sound. If a selection exists, the script puts the RECITE 
string into a temporary variable. The "RAVE "{"and "}"" are stripped from 
the temporary variable so that only phonetic/timing values remain. The 
script then finds out how time exists before the selection. The script 
then finds out how much time is in the selection and checks to make sure 
the selection is valid. If the selection is not valid, the user is warned 
and the script stops executing. The script then builds a RAVER "FIRST" 
command by including the time before the selection, the time in the 
selection, and the selection itself. The RAVER "FIRST" command makes the 
Navigator speak the selection. 
If there is no selection, the script in the Test 3 button examines the Talk 
Command field to make sure it contains text. If there is no text, the user 
is warned and the cursor is placed in the Talk Command Field. 
The script of the Test 3 button then tests the RECITE command in the Talk 
Script field to make sure it follows correct syntax. If the RECITE command 
in the Talk Script Field is not correct, the script warns the user and 
stops executing. The RAVE RECITE command is then sent to the RAVE driver 
where which presents the sound and animation. 
If text is selected within the Text String field and the return key is 
pressed, the hyperanimator system will highlight the corresponding word in 
the Phonetic String Field. The script of the Text String Field first 
checks to make sure the Field is not empty. If the Text String Field is 
empty, the user is warned and the script stops executing (15 of Appendix 
III). The script of the Text String Field determines the position of the 
selected word and selects the identical word position in the Phonetic 
String Field (16 of Appendix III) 
If text is selected within the Phonetic String field and the return key is 
pressed, the hyperanimator system will highlight the corresponding 
phonetic/timing value pairs in the Talk Script field. The script of the 
Phonetic String Field first checks to make sure the Field is not empty. If 
the Phonetic String Field is empty, the user is warned and the script 
stops executing. 
The selected text is placed in a variable and unneeded stress numbers 
associated with the text are removed from the phonetic string. The script 
then places the RAVE RECITE string into a temporary variable called 
tempString. The "RAVE "{RECITE", SoundName, and "}"" are removed from the 
string and stored in another temporary variable so that tempString 
contains only the phonetic/timing value pairs. A string containing only 
the phonemes in the RAVE RECITE string is then constructed. 
The selected text is then matched up with the corresponding phonemes in 
tempString and the starting and ending point of the selected text are 
marked. If no match was found, the flag FirstStart will contain 0, the 
user is warned and the script stops executing. Because the starting and 
ending points indicate phonemes, the number of characters therebetween 
must be doubled because the final selection will comprises phoneme/timing 
value pairs. The final selection produced by the script is made by 
character position within the field. To determine which character to start 
at and which character to end at the total number of characters in a 
variable called introString is determined. Five is automatically added to 
this number to provide for missing spaces and quotes. The total length of 
the phonetic/timing value portion which was stored in tempString is then 
found. The total length of the selection of phonetic/timing value pairs is 
then found. The length of the introString plus the length before the 
selection marks the position of the first character in the string to 
select. The beginning character position plus the duration of the 
selection defines the position of the last character to select. 
If the return key is pressed while the cursor is within the Talk Script 
Field, the hyperanimator system will send the RAVER "SECOND" command which 
instructs the RAVE driver to recalculate the RECITE String from the 
cursor's insertion point to the end of the string. If a stop (bullet) 
character is present within the RECITE String, then only the 
phonetic/timing value pairs between the cursor position and the stop will 
be recalculated. The position of the cursor is first determined and then a 
validity check is made. If a stop character is in the RECITE string, it 
must not be too close to the cursor insertion point to prevent a proper 
recalculation of the selection. 
The phonetic/timing value pairs between the cursor and the stop character 
are then determined and the timing values before the cursor summed. The 
timing values after the stop character are added. A RAVER "SECOND" command 
is constructed with the totalTime of sound and the string between the 
cursor and the stop character. The RAVER "SECOND" command returns the 
modified RECITE command and it is displayed in the Talk Command field. 
If there is no stop character, the entire string after the cursor will be 
recalculated. First, the selected string to be recalculated is created and 
displayed after the cursor and the timing values for the selected string 
are summed. A RAVER "SECOND" command is then constructed with the 
totalTime of sound in the string. The RAVER "SECOND" command returns the 
modified RECITE command and it is displayed in the Talk Command field. 
When the Speech Sync Lab is closed (i.e., the synchronization process has 
been completed and the user has transferred to another screen), the RAVE 
commands "CLOSESOUNDFILE" and "UNLOCK" are issued to close the sound 
resource file that has been opened and place it into memory. These RAVE 
commands are issued only if a sound has been processed. The variable 
soundFlag will contain "true" if a sound has been processed. 
The RAVE driver comprises two parts each having different functionality. 
The first driver/functionality is editing of synactors, and editing of the 
sound synchronization. The second driver/functionality is to bring life to 
a synactor. Commands for the speech synchronization process are listed in 
Table 2, below. 
TABLE 21 
__________________________________________________________________________ 
RAVE Scripting Language Commands 
Name Parameters Return Value 
Other Action 
__________________________________________________________________________ 
ACTOR name of actor, coordinate location 
none (RAVE) 
ACTORINFO none actor information 
CLOSEACTORFILE 
none none (RAVE) 
CLOSESOUNDFILE 
none none (RAVE) 
CONVERT text string phonetic string 
COPY new/next image number 
none (RAVE) 
display image 
DIGIMAKE sound name, string of phonetic/timing 
recite string 
value pairs, 
EDITINFO none actor information 
EXPRESS image(s) none (RAVE) 
FIRST starting point, length of sub-sound, 
none (RAVE) 
talks 
phonetic/timing value pairs 
FREEZE none none (RAVE) 
HIDE name of actor none (RAVE) 
INTERMISSION 
none none (RAVE) 
LOCK none none (RAVE) 
MOVE coordinate location none (RAVE) 
OPENACTORFILE 
none name of actor 
dialog 
OPENSOUNDFILE 
none name of sound 
dialog 
PASTE current image, number of total images 
none (RAVE) 
PHONETIC text string none (RAVE) 
PITCH integer value none (RAVE) 
RECITE sound name, phonetic/timing value 
none (RAVE) 
talks 
string 
RETIRE name of actor none (RAVE) 
SECOND total before adjust point, phonetic/timing 
pho./timing pairs 
pairs 
SHOW name of actor none (RAVE) 
SPEED integer value none (RAVE) 
STATUS none test string 
SYNCSOUND name of sound none (RAVE) 
UNFREEZE none none (RAVE) 
UNLOCK none none (RAVE) 
USE name of actor none (RAVE) 
.vertline..about.SPEED 90.about..vertline. 
none none (RAVE) 
__________________________________________________________________________ 
COPYSOUND: same as copyactor but for sounds 
DELETEACTOR: removes the actor after asking user which file to remove it 
from. 
DELETESOUND: same as deleteactor but for sounds. 
CURRENT: lets the editor know what the name of the current actor is that 
has been brought in by the runtime driver 
SAVE: saves the current actor to disk. 
REVERT: reverts the current actor to the last one that was saved to disk. 
GO TO IMAGE: causes the image given to become the displayed image. 
NUMFACES: returns the number of faces and phonemes in the current actor. 
SIZE: returns the height, width, and top left position of the current 
actor. 
PASTE: takes the current picture and makes it part of the current actor. 
COPY: takes a picture from the current actor and puts it in the clipboard 
COMPILE: creates an actor from a pho file and an image file. 
ACTOR: brings the specified actor into memory. 
RETIRE: removes the specified actor from memory 
EXPRESS: animates the actor by showing the given expression 
MOVE: moves the actor to the location specified 
HIDE: hides the actor. 
SHOW: shows the actor. 
RECITE: the actor will speak the given information while the sound is 
played. The recite command is in phonetics not image indices so it will 
work with any model. 
"text": the actor will speak the text using a speech synthesizer. 
PITCH: adjusts the pitch of the speech synthesizer. 
SPEED: adjusts the speed of the speech synthesizer. 
PHONETIC: the actor will speak the phonetic string using the speech 
synthesizer. 
FREEZE: lock the actor to its current position so that may not move when 
clicked on. 
UNFREEZE: allow the actor to be moved by clicking and dragging. 
LOCK: Sets a flay that requires the actor to remain in memory. 
CONVERT: Takes a line of text as its parameter and returns the 
corresponding phonetic string. 
OPENSOUNDFILE: Allows the user to select the digitized sound that this 
text corresponds to. Returns the name of the sound. 
SYNCSOUND: takes the name of the sound from opensoundfile as its paramete 
and anlyzes that sound for length and other characteristics. Establishes 
that sound as the sound to be used with the Digimake command. 
DIGIMAKE: takes the sound name and the phonetic string that was the resul 
of the Convert command as parameters. Returns the Recite command string 
that the user could then use to call rave and have their sound spoken. 
RECITE: In the context of speech sync, the recite command is used to test 
the Recite string that resulted from the Digimake command. 
FIRST: The First command will "recite" only the selected portion of the 
Recite command string. 
SECOND: The Second command will assist the user in maintaining the correc 
total of timing values by recalculating a portion of the recite command. 
UNLOCK: Unlocks the RAVE actor and allows it to be removed from memory. 
CLOSESOUNDFILE: Closes the currently open sound file and removes it from 
the known sound position. 
OPENACTORFILE: opens a file with an actor. 
CLOSESOUNDFILE: closes the currently open actor file. 
COPYACTOR: asks the user for the file from which to copy the actor, and 
the file to which to copy the actor, then does it. 
INTERMISSION: remove the actor, and the entire driver from memory. 
STATUS: returns information about the driver and its living conditions. 
The RAVER Command DIGIMAKE takes the sound name and the phonetic string 
that was the result of the CONVERT command as parameters. It returns the 
Recite command string that a user can then use to call RAVE and have the 
sound spoken with synthesized animation. 
A preferred embodiment comprising a relatively simple design is described 
below. Other designs utilizing known methods of speech recognition could 
be utilized either alone or in combination with the below described 
phonetic proportionality method. A desired prerecorded, digitized sound 
resource is called up and its length in time ticks calculated and stored. 
Then, utilizing a phonetic string that corresponds to that recorded sound, 
selected text from that sound resource is converted to a list of phocodes. 
The phocodes are then looked up in a table of relative timing values which 
provides a value for how long the associated face or position image for 
each phonetic code is to be used. The table can be coded in the program or 
generated from the RAVEL file using extensions to the RAVEL language to be 
unique to a synactor model. It can thus be used with synactors with 
varying accents, drawls and other speech mannerisms or languages. An 
example of such a table is shown in Appendix IV. During the speech 
synchronization process, this table is utilized to look up timing values 
for each phocode. Each line in the table represents a phocode and its 
associated relative timing value. The first line is all null characters 
and is used as a place holder so that the indexing of phocodes will be 
useful numbers. The first character is the first letter corresponding to 
the phocode, the second character is the second letter corresponding to 
the phocode, if there is one, or an end of string character. The third 
character in each line is an end of string character for the two letter 
phocodes or a space filler. The fourth character is the relative timing 
value associated with that phocode. The last line is again all null to 
mark the end of the table. 
Once the two parallel lists, phocodes and relative timings, and the length 
of the associated sound have been established, the actual process of 
synchronizing the speech to the sound is initiated by refining the timing 
list. This process is illustrated in FIG. 27. The first step is to figure 
the sum of all the values in the timings list. This sum is then compared 
to the sound length. The timing value given to each phocode is then 
adjusted proportionately with the compared sums and rounded to whole 
numbers. Also, if the total of the timings is less than the sound length, 
then the timings are decremented until the total of the timings matches 
the total sound length. If the total of the timings is greater than the 
total sound length, then the timings are likewise incremented until they 
match. This is done to deal with cumulative rounding errors because the 
timings must be integer values so the RAVE real time phase can operate. 
If the timing value of the first phocode is within a certain small range, 
then we split it in half and distribute the resulting amount throughout 
the other timings. If the timing value is larger than this, we would still 
decrement it by some, and distribute that to maintain the total of the 
timings being equal to the sound length. Any phonetics that may have a 
zero timing value are removed from the list. 
The result is a list of phocodes and timing values which represents the 
synchronization of the faces to the corresponding sound. To create a 
Recite command, the phocodes are used again to look up the corresponding 
phonetics. The Recite command will coordinate the actual sound/motion 
combination. The user can edit the command on the screen to fine tune it, 
test it, and edit it more until it looks satisfactory. (Editing is 
particularly useful for unusually-timed speech segments, for example, with 
one word pronounced more slowly or differently, or with silences or throat 
clearings not reflected in the text and/or not amenable to speech 
recognition.) To help a user fine tune the Recite command, the 
hyperanimator system provides methods to isolate, test, and/or 
programmatically resynchronize individual portions of the sound and 
animation to fine tune each by itself. The "FIRST" command uses its 
parameters, starting point, length of part to be played, and the 
associated phonetic/timing pairs, to determine which part of sound to 
play, and then uses the Recite technique to play that subsound with the 
associated phonetics and their timings. The "SECOND" command strips the 
previous timing values from the parameter string of phonetic/timing pairs 
and employs the same system as the Digimake, except that the sound length 
is decremented by another parameter and the amount of time not included. 
For example, the text "the quick brown fox" (1 in Table 3) converts to the 
phonetic list of "DH, AX, K, 2, IH, K, B, R, OW, N, F, AA, K, S: (2 in 
Table 3). After looking these up in the timing table, a timing list of 
"10, 6, 6, 7, 6, 6, 7, 7, 10, 7, 7, 10, 6, 7" (3 in Table 3) is specified 
for a total of 102 (6 in Appendix VI). If, for example, the speaker has a 
drawl and the sound length is 105 (5 in Table 3). The result of running 
through the timing adjustment routine would be "DH 4 AX 7 K 7 W 8 IH 7 K 7 
B 8 R 8 OW 9 N 8 F 8 AA 9 K 7 S 8" (4 in Table 3). 
APPENDIX VI 
______________________________________ 
##STR1## 
##STR2## 
##STR3## 
______________________________________ 
To create more natural animation, the RAVE driver includes facilities to 
handle variations in facial positioning that occur as a result of 
coarticulation. Coarticulatory patterns of speech exist when two or more 
speech sounds overlap such that their articulatory gestures occur 
simultaneously. To some extent, this affects practically all natural 
speech sounds. A major effect is of the lip, jaw, and tongue position of 
various vowels on the articulator's position for consonant production. A 
dramatic example of this is to compare the lip configuration in forming 
the consonant "b" when it is in the following vowel environments: "eebee" 
vs. "ooboo". There are two major types of coarticulation, both of which 
are operating at the same time. Inertial Coarticulation is the result of 
the articulatory apparatus (i.e., lips and face) being a mechanical 
system, i.e. "mechanical slop". The articulator positions for a previous 
sound are retained and affect the articulator positions of the next sound. 
Anticipatory coarticulation is the result of neural control and 
preplanning for increased efficiency and speeds of articulator movement, 
i.e., natural articulators are controlled in parallel. The articulator 
positions for the target sound are affected by the anticipated positions 
for the next sound. 
The RAVEL language is able to handle coarticulation in several ways. The 
basic method is to provide for the definition of a number of coart types. 
In most languages, for lifelike animation of synactors three coart types 
are sufficient: SILENCE, VOWELS, and CONSONANTS. Certain coart variant 
groups may also be defined. Utilizing these three types in a RAVEL 
program, the number immediately following the VOWELS command tells the 
RAVEL compiler the number of coarticulatory groups for the specific 
synactor and language which is being described. This sets the valid range 
for all COART commands. For example, in English, for simple models, three 
coarticulatory groups are typically defined: neutral, retracted, and 
protruded. The valid coarticulation values then are 1, 2, and 3. 1 will 
always be the default coarticulation variant for any language. The user 
can choose which group the 1 will represent. 
When the CONSONANTS command is issued in RAVEL, all following phonemes will 
be assigned the coart type "c", for consonant. The valid COART values are 
determined by the VOWELS command as described above. 
To build the coarticulations table a third operation code has been added to 
the variable language called COART. For every image and timing 
synchronization pair there must also be a COART group assigned. This 
assignment is built into the coarticulations table at the same time that 
the synchronization pair is built so that there is an explicit one to one 
relationship between them and that their phocode will reflect this. 
All coarticulator variants for any particular phoneme are grouped together 
in the ravel program in the group sequence order. The first variant is 
considered the default coarticulatory synchronization. The second through 
n-th variants are the second through n-th coarticulatory synchronizations 
for the specific phoneme being defined. 
An example of a coarticulated model is given in Appendix V. It is an 
example of a portion of a RAVEL file that describes the coarticulatory 
relationship to the image and timing value of phonemes. 
Referring now to Appendix V, the coart types are defined. The illustrated 
model has three coart types: SILENCE, VOWELS AND CONSONANT. The word 
"SILENCE" (1), indicates that the following RAVEL lines of phoneme 
definitions are silent coart types that have no action or sound. The 
defined silent phonemes (2) follow "SILENCE". The word "VOWELS" (3), 
indicates that the following RAVEL lines of phoneme definitions are vowel 
coart types that are used to determine the coarticulatory variant of 
preceding and following consonants to be used in any syllable. The number 
"3" (4), indicates the number of coarticulatory variant groups to be used 
by the language being defined. In the case of English there are three 
groups: retracted, neutral, and protruded. The phoneme definitions (5) 
follow "VOWELS". The columns (6) indicate the pronounciation timing and 
image codes for the VOWEL phonemes. Column 7 indicates the new 
coarticulation variant group indicator. The coarticulation variant group 
indicator can have a valid value from 1 to n, n being determined by the 
number (4) of coarticulation variant groups for the language being 
defined. The word "CONSONANT" (8), indicates that the following RAVEL 
lines of phoneme definitions are consonant coart types. For every 
consonant coart type phoneme there are n phoneme definitions (9), n being 
determined by the number (4) of coarticulation variant groups for the 
language being defined. Each of these phoneme definitions for a particular 
phoneme represent a unique variant of the consonant for one of the 
coarticulation variant groups. "SILENCE" (10) indicates that the following 
RAVEL lines (11) of phoneme definitions are silent phonemes. Because these 
expressions have image and timing values (12) defined they also belong to 
a coarticulation variant group. Coarticulation variant group 1 (13), is 
the neutral variant group and should be used for most expression images. 
Referring again to FIG. 8, in the synactor model table 810,, Coart types 
points to the coarticulation type table. This is a count off table, 
counted off by phocode, having a filler entry for phocode zero each entry 
of which specifies which coart type. The number of types depends on the 
model and is input in RAVEL source code. In English three is an 
appropriate number of types. Each corresponds to whether a phocode is a 
V(OWELS), C(ONSONANTS), or S(ILENCE). [In the previous patent application 
V(OWELS) are equivalent to VOWEL EVENT, C(ONSONANTS) are equivalent to 
EVENT not preceded by VOWEL, and S(ILENCE) is equivalent to non events.] 
The coarticulation type is used to determine what effect the particular 
phocode will have on its neighboring phocode in a give string. It is 
generated by compiling RAVEL source code describing that synactor model. 
At runtime, in the enhanced synactor model table, Coarticulations points to 
the coarticulation table. This is a count off table, counted off by 
phocode, having a filler entry for phocode zero, separated by a dedicated 
code, each entry of which specifies the coarticulation group corresponding 
to each position and timing pair component as given for that phocode in 
the RAVEL source file. 
The coarticulation values range from 1 to n, n being defined by the RAVEL 
VOWELS command as the number of coarticulation groups for the particular 
synactor. This value is added to the phocode when looking up position and 
timing sequences in the sequence table to get the correct sequence for the 
particular coarticulation in progress. 
The process whereby a coarticulated model is determined and applied by RAVE 
is as follows. First, using the existing text to phonetics converter a 
text string is converted into a phocode string. This phocode string is 
decomposed into its string of syllables. To break words into syllables the 
following rules are sufficient: A syllable contains one vowel surrounded 
by zero to n consonants on either side. Several simplifications are 
sufficient to create good coarticulated animation in a three-type, 
three-group scheme. It is sufficient to assign the coarticulator variant 
of a consonant that is between two other consonants as that consonant's 
neutral coarticulatory variant. The coarticulatory variant of a consonant 
that is between a consonant and a vowel uses the coarticulatory variant as 
determined by the vowel it is next to. The coarticulatory variant of a 
consonant that is between two vowels is determined by the vowel following 
the consonant. More elaborate methods could be constructed to provide much 
more accurate realism, but are not necessary for this simplified case. 
Each syllable is then decomposed into its phocodes. Each phocode is then 
used to look up its coart Type, (whether it is a vowel, consonant, or 
silent). Silent phocodes have no coarticulatory sequence and are ignored. 
All consonant phocodes preceding the last consonant phocode component 
immediately preceding the vowel are assigned their neutral coarticulatory 
synchronization pair. The final consonant phocode component immediately 
preceding the vowel is determined by the first phocode component of the 
vowel (anticipatory coarticulation). The coarticulation value of the first 
phocode component of the vowel is looked up in the coarticulation table. 
This value minus one is then added to the phocode of the preceding 
consonant. This new phocode value is then used to lookup the preceding 
consonant phocode's final component's adjusted coarticulatory 
synchronization pair. The synchronization pairs for all components of the 
vowel are then looked up and added to the list. The first phocode 
component of the consonant immediately following the vowel is determined 
by the last phocode component of the vowel (inertial coarticulation). The 
coarticulation value of the last phocode component of the vowel is looked 
up in the coarticulation table. This value minus one is then added to the 
phocode of the following consonant. This new phocode value is then used to 
lookup the following consonant's first component's adjusted coarticulatory 
synchronization pair. All consonant phocodes following the first consonant 
phocode component after the vowel are assigned their neutral 
coarticulatory synchronization pair. 
Referring now to FIGS. 11a-11d, animation sequences with (11c and 11d) and 
without (11a and 11b) coarticulation are illustrated. It demonstrates how 
coarticulation helps make talking synactors always look their best. Note 
that the middle positions 1101 for "B" are both the same in 11a and 11b, 
but are different in the "OOBBOO" of 11c and 11d (positions 1103 and 1105, 
respectively). This contrast between the coarticulatory effects of the 
retracted vowel "EE" and the protruded vowel "00" on the consonant "B" 
occurs as follows. 
The text string "EEBBEE" is decomposed using the existing text to phonetics 
converter into its phocode string. Each phocode is used as an index to the 
coart type table to lookup whether it is a vowel, consonant or silent. The 
first phocode in the phocode string representing the vowel "EE" has a 
coart type of `V`. Because there are no consonants before it, no 
anticipatory coarticulation occurs. The image and timing pair for "EE" is 
selected and placed at the beginning of the synchronization list. 
The second phocode in the phocode string representing the neutral variant 
of the consonant "B" has a coart type of "C". Because it immediately 
follows the vowel "EE", inertial coarticulation occurs. The coarticulation 
for the last component of the vowel "EE" is 2, representing the retracted 
coarticulation group for the English language. This coarticulation group 
minus 1 is added to the neutral variant of the consonant "B"'s phocode 
results in the coarticulatory adjusted phocode for the consonant "B". The 
coarticulatory adjusted phocode for the consonant "B" is then used to look 
up the image and timing pair which is added to the end of the 
synchronization list. 
The third phocode in the phocode string representing the neutral 
coarticulatory variant of the second "B" also has a coart type of "C". 
Looking ahead, the program determines that the fourth phocode in the 
phocode string representing the second occurrence of the vowel "EE" has a 
coart type of "V". This will have an anticipatory coarticulatory effect on 
the second consonant "B" that precedes it. The coarticulation group for 
"EE" is 2, representing the retracted coarticulation group for the english 
language. This coarticulation group minus 1 is added to the neutral 
variant of the consonant "B"'s phocode resulting in the coarticulatory 
adjusted phocode for the consonant "B". 
The coarticulatory adjusted phocode for the consonant "B" is then used to 
look up the image and timing pair representing the consonant "B" when it 
is immediately affected by a retracted vowel which is added to the end of 
the synchronization list. Finally, the image and timing pair for the 
fourth phocode in the phocode string representing "EE" is added to the 
synchronization list. The images 1103, 1104 in FIG. 11c show the results 
of processing "EEBBEE". 
In contrast, the same process for "OOBBOO" would result in the set of 
images 1105, 1107 of FIG. 11d because the vowel "OO" belongs to the 
coarticulation group 3, representing the protruded vowels of the English 
language. The coarticulation group 3 minus 1 is added to the neutral 
variant of the consonant "B"'s phocode resulting in the coarticulatory 
adjusted phocode for the consonant "B" which represents the consonant "B" 
when it is immediately affected by a protruded vowel. Without the 
coarticulation process being applied to the synchronization selection 
process the neutral variants of "B" are used which means that both "EE" 
and "00" will be displayed using the same image form for the consonant "B" 
resulting in the images 1101, 1102 and 1101, 1106 shown in FIGS. 11a and 
11b, respectively. 
##SPC1##