2. Interactive Performance of Cindy

A Virtual Dancer "Cindy"

Overview
Two players
improvise on different
musical instruments

Interactive Performance ofa Music-controlled CG Dancer

Cindy is controlled andchoreographed bymusical performance

School of Science and Engineering, Waseda University

Masataka Goto
1996/10/10 LCC96

No.1

CGCG
Sound
Sound

Player A
Drummer

Player A (Drums)
3D CG

Player B
Guitarist
PianistNo.5

1. Introduction

Multimodal Interaction

SoundInteractive Performance

Cindy does not movein time to the music
autonomously

Yoichi Muraoka

Interactive Performance

CG Dancer
SoundCRRIHT

Seven kinds of drum-soundsMusicSDInteraction between
Two Players

Virtual dancer enables
two players to interact
through music and 3D computer animation(auditory and visual information)

Bass Drum
Snare Drum

(BD)
(SD)

Basic Motions

Low TomMiddle TomHigh Tom(LT)
(MT)
(HT)

Accent Motions

Crash Cymbal (CR)Ride Cymbal (RI)

Showy Motions

MTBDLT

Each is mapped to a different dance motion
Our Goal
Control the timing of each dance motion
by the onset time of the drum-soundAchieve a new mode of interaction in which
CG and music are closely integrated by a LCC
No.2

No.6

Previous Performance

Player B (Guitar, Piano, etc.)
Parameters periodically derived from performance

Interactive system between a player and a LCC

NumNotes: The number of notes
AvePitch:
Average pitch
ChordMeas: Whether a single note or chord is played

Interaction between human players
was not considered

Switch six predefined sets of mapping
between drum-sounds and dance motions

Conventional Jam Session

1) Normal motions

Default2) More cheerful motions

NumNotes: Many

3) Quiet and small motions

NumNotes: Few

auditory information

4) Hand-raising motions

AvePitch: High

auxiliary visual information(gestures)

5) Dynamic motions

ChordMeas / NumNotes: Large

6) More dynamic motions

ChordMeas / NumNotes: Larger

Musical interaction between players is essential
Exchange

No.3

No.7

Virtual Dancer "Cindy"

Mapping between drum-sounds and motions
Player A

Enhance visual interaction in a jam session
Provide two playerswith more expressive way to communicate

RICRHTMT

Two players
Choreograph Cindy together
by their musical improvisation in real time
Play different roles in choreographing

No.4

Waiting poseLTSDBD

Drum-sounds

Cooperation between players is important
to make music goodto make the dance full of variety

Player B

MotionMotion mapping 1
Motion mapping 2
Waiting pose
Motion mapping 6

No.8

3. Three Kinds of Interaction

4. Implementation

Interaction between players through music

Distributed Computing Environment
SGI Indigo2 Extreme x 2

Musical interaction

Ethernet

CG Dancer

MIDI

Timing

Improvise while
reacting to eachother’s performance

Mapping

Cooperatively try to
make music good

RMCP (Remote Music Control Protocol)

Cooperative
Interaction

DirectInteraction

Communication protocol on the UDP/IP
based on the server-client model
MIDI data of players’ performances are broadcast as RMCP packets

Indirect
Interaction

Player A

MusicalInteraction(Musical Instrument Digital Interface)

Allocate different tasks on several computers

Player B

No.9

Achieve good load-balancing

No.13

Interaction between each player and Cindy

RMCP Servers and Clients

CG Dancer

MIDI Instrument

TimingDirect interaction(Player A)

Mapping

DirectInteraction

Cooperative
Interaction

Determines the timing of dance motion

Player A

MusicalInteraction

CG Dancer

MIDI

Indirect
Interaction

RMCP Sound Server

RMCP Animation Server

Player B

RMCP Music Analyzer

Directly changes Cindy
while looking at the dance motion

RMCP MIDI Station

Cannot determine the motion
mapped to each drum-sound

RMCP MIDI Station

Player AororDrums

RMCP MIDI Receiver

RMCP MIDI Receiver

MIDI

1. Take each player’s performance as input

maintain the rhythm in music
determine the timing of dance
No.10

Player B
PianoGuitar

MIDI

RMCP MIDI Receiver
RMCP MIDI Station

2. Output sounds of players’ performance

RMCP Sound Server

3. Analyze player B’s performance

RMCP Music Analyzer

4. Display Cindy on 3D computer animation

RMCP Animation Server

No.14

5. Experimental Results

Interaction between each player and Cindy
CG Dancer
TimingIndirect interaction (Player B)

Conditions

Mapping

DirectInteraction

CooperativeInteraction

IndirectInteraction

Switches the mapping set

Player A

MusicalInteraction

Improvised music such as rock, pop, and fusion

Player B

Player A played the drums on a synthesizer keyboard

Indirectly changes Cindy
while confirming the change of the mappingby looking at dance motions made by A

Player B played a melody or chords on a MIDI guitar

Results
Choreographed Cindy directly and indirectly by improvising

Cannot make Cindy move by himself

To vary Cindy’s motions
they tended to make performance more variedthan performance in a conventional session

characterize the mood in music
determine the mood of danceInteracted visually to change the motions cooperatively
No.11

No.15

Interaction between players through Cindy

Video Explanation

CG Dancer
TimingCooperative interaction

Mapping

DirectInteraction

Cooperative
Interaction

Indirect
Interaction

Choreograph Cindy cooperatively
to make the dance full of variety

Player A

MusicalInteraction

Player B

Player A
Drum-sounds

Player A tries to change motions
in a manner appropriate to
the current mapping that player B selectsPlayer B tries to change the mappingin a manner appropriate to
the way that player A currently plays
No.12

Player B

CRRILTMTBDSDMotion mappingHT

Currently played
drum-sounds
No.16

456

12

3

Currentmapping set

Examples of Cindy’s motions

No.17

6. Conclusion
Summary
Interactive performance system in which
two players and Cindy interact through music
Three kinds of interaction
1. Interaction between players through music
2. Interaction between each player and Cindy
3. Interaction between players through Cindy
Implemented on distributed workstations
Achieved an interesting new interactionthat cannot be achieved through music alone
No.18

Design of Interactive Performance
Avoid mismatches between media and keep harmony
Perform three kinds of interaction simultaneously
without feeling disharmonious
Use thedrums, which mainly maintain the rhythm in music,
guitar, which characterizes the mood in music,

in thedirecttimingof danceinteraction determining theindirectmood

Consider harmony between the dance motions andthe mood of the improvised performance
Cheerful
cheerfulmotions are selected when performance isQuiet
quiet
No.19

Future Work
Design other interactive performancesInteractive performance system for three or more players
for other LCCs

Remote interactive performance
Our implementation facilitates
use of the system in settings where
the players are not in the same physical location

Interactive performance via InternetNo.20