LCC’96: Abstract Submission

A Virtual Dancer ‘‘Cindy’’

Interactive Performance of a Music-controlled CG Dancer
Masataka Goto and Yoichi Muraoka
School of Science and Engineering, Waseda University.

3-4-1 Ohkubo Shinjuku-ku, Tokyo 169, JAPAN.

fgoto, muraokag@muraoka.info.waseda.ac.jp

Various lifelike computer characters have been proposed in recent years [LCC’94, LCC’95]. In this paper, we introduce
an innovative way of using a lifelike computer character to enhance the human-human and human-computer communication.
We propose an interactive performance system in which a virtual CG dancer enables two players (musicians) to interact witheach other not only through music but also through 3-D computer animation. The players not only improvise together as in aconventional jam session, but also observe the virtual dancer, whose motion is changed in real time according to their musicalperformance. In other words, they can communicate using both auditory information and visual information. The purpose of thisresearch is to achieve a new mode of interaction in which CG and music are closely integrated by a lifelike computer character.
Interaction between a player and a lifelike computer character through musical performance was reported [Goto, 1993], butinteraction between players was not considered in those reports. In real jam sessions, musical interaction between players isessential and is achieved by exchanging not only auditory information but also auxiliary visual information such as gestures thatindicate repetition or the end of song.
To enhance the visual interaction in a jam session and provide players with more
expressive way to communicate, we introduce a virtual dancer called Cindy visible
to both players. Two players, a drummer and a guitarist (or pianist), can choreograph Cindy together by their musical improvisation in real time. The players canuse three kinds of interaction at the same time: 1) interaction between the players
through their musical sounds, 2) interaction between each player and Cindy, 3)interaction between the players through Cindy. The first interaction is the musical interaction of a conventional jam session, and the second interaction is both
direct and indirect. In the direct interaction the drummer determines the timing ofa Cindy’s motion, since each drum-sound is directly mapped to a Cindy’s motion.

CG DancerTiming
Mapping

DirectInteraction

CooperativeInteraction

Indirect
Interaction
MusicalInteraction
Player A Drummer

Guitarist Player B
PianistThere are six predefined sets of mappings between seven kinds of drum-sounds
and Cindy’s motions, each for a different mood of dance motions. In the indirect interaction the guitarist switches these mapping sets by changing a mood of his musical improvisation. If the guitarist plays, Cindy does not move unless the drummer
determines the motion timing; information from both players is necessary to choreograph the dance totally. The third interaction,
cooperation between the two players, is therefore important in creating a complete performance.
This lifelike computer character system has been implemented on distributed workstations connected by the Ethernet. MIDI
data of two players’ performance is sent to a graphics workstation that displays Cindy and to a workstation that controls a MIDI
instrument that produces sounds. In our experimental setup, the three kinds of interaction were performed on the system, andthe multimodal interaction produced very satisfactory results.
Relevant Categories:

multi-media/multi-modal interfaces, computer graphics, audio interfaces