A Jazz Session System forInterplay among All Players

2. Jazz Session Model for Interplay
Jazz Session ModelAll players canlisten and react toall other players

VirJa Session

Player 1

Player 2

Human

Human

ComputerComputer(Virtual Jazz Session System)
Interact without fixed
leader-follower
relationship

School of Science and Engineering, Waseda University

Masataka Goto

Isao Hidaka

Yosuke Kuroda

Hideaki MatsumotoComputer players
play a kind of solo

Yoichi Muraoka
1996/08/23 ICMC96

No.1

Interaction
Player 3

Player 4

Human

Human

ComputerComputer

No.5

1. Introduction

Imagine: All Players are Human Players
Useful for remote sessions in which
players are not in the same physical location

What is Important in Jazz Session?
Interplay
All players improvise together
while reacting to other players’ performanceswith no leader-follower relationshipExchange musicalinformation throughcomputer networks

Multimodal Interaction
Communicate by musical sounds andadditional visual information such as gestures
No.2

Player 1

Player 2

Human

Human

ComputerComputerInteraction
Player 3

Player 4

Human

Human

ComputerComputer

No.6

Imagine: All Players are Computer Players

Our Goal
Build a computer system that enables
interplay among humans and computers

Effective when various designers implementdifferent computer players with various
characteristics
Player 1
Player 2

Simulate the actual interactionthat occurs among human players
Designers’ substitutes
interact witheach other

No.3

Human

Human

ComputerComputerInteraction
Player 3

Player 4

Human

Human

ComputerComputer

No.7

Related Work

Jazz Session Model for Piano Trio

Jazz session / Automatic accompaniment( Nishijima et al., 1992
Dannenberg, 1984
Rowe, 1993

Kondo et al., 1993
Vercoe, 1984
Horiuchi et al., 1993

Computer players
are executed
as separate
processes

Wake et al., 1994
Baird et al., 1989
Hidaka et al., 1995 )Player 1: Pianist
Human
ComputerGenerate several accompaniment parts together

Player 2 listens
to Player 3
as well as
Player 1

Only react to the soloist’s performance
Fixed leader-follower relationship

Mention interaction among computer players

Player 2 and 3interact witheach other( Kanamori et al., 1993 )Only focus on the music-listening process
Not generate musical performance
No.4

No.8Interaction

Interaction

Virtual Jazz Trio
Player 2: Bassist

Player 3: Drummer

Human

Human

ComputerInteraction

Computer3. VirJa Session (CG and Camera)

4. Computer Bassist and Drummer
Piano Drums (Bass)

VirJa Session

Music Listening

Player 1: Pianist

VirJa Session

Music Listening

Human

Detect Musical Primitives
Extract Intention Parameters

ComputerMusical primitives

All players
communicate bothby listening toother players

CaalDetermine His Own Leadership
Percentage based on Scenario

Tension, Chord emphasis
Chord substitution, Theme reprise

HumanPerformance Improvising

Computer

Virtual Camera

Leadership percentage
Highest0 Intention Param. 100

Real-time CG

Predict the Next Leadership
Percentage for Other Players

Excitement, Rhythm emphasis,

Player 3: Drummer

raRe

me

ComputerCa

Humanalmera

Virtual Jazz Trio
Player 2: Bassist

Session Understanding

Intention parameters

Reby seeing
each others’
bodies andgestures

Estimate Leadership Percentage

Chord note, Tension note, Scale note
Substitute chord note, Theme note,
Loud note, High note, Many notes

Virtual Camera

PianoDrums

Choose Pitch/Loudness/Rhythm
Patterns from Databases

Bass

Combine Patterns

Real-time CG

55%

10%35%

Leadership Percentage

No.9

Bass (Drums)

No.13

Multimodal interaction among all players
using both auditory and visual information
Human Player:
Computer Player:

See computer players on CG
See human player by camera
See computer player via network

For players:
Feel the presence of other playersas if they were actually playing together
Cooperate well through visual information

No.10

Consider the wholemusical relationships
among players

For audiences:
Feel the presence of all players
as if they were attending a live concert

Visualization of Computer Players on CG

Determine how muchhe tries to leadthe session

1 Predict1

Select databases

100-30= 70%
Target2 Leadership

Percentage(based on Scenario)

30%

13
2

4
4

3

5100

5

Hard Pre-registeredSoft

Pattern Databases

Combine patterns
Pitch

No.14

5. Implementation

LoudnessRhythm

B Beat Information
A Animation InformationM Music Information
G Gesture Information

VirJa Session

Player 1: Pianist
Human
Computer

SGI Indigo2
Impact x 1
Extreme x 2(2) Keeping time to musical beatsby foot-tapping or rocking his body(3) Making two kinds of gestures

M

Ethernet(5) Turning his eyes to another player

Ethernet
Player 2: Bassist

B

Human

RMCP

Computer

M
G M
G B
A

RMCP Player Server(Remote Music Control Protocol)

Song form: combination of song parts

RMCP Beat Provider

Communication protocol on
the UDP/IP based ontheme (reprise)M

PianistGMIDI

Scenario

RMCP Sound Server

RMCP MIDI Receiver

RMCP Camera Analyzer(4) Nodding to show that he understood a gesture

No.11

70%

320

Environment

piano solo
bass solo
four verses (piano / drums)

Leadership
Percentage

Performance Improvising(1) Playing musical instruments according to his sounds

the server-client model

M
B
A

RMCP Animation ServerBMBAM
M GBGA

Player 3: Drummer
Human
Computer

RMCP Player Server
RMCP Beat Provider
RMCP Animation ServerBassist

Drummer

No.15

Multimodal Interaction

RMCP Servers and Clients

The number of repetitions of each song partis determined dynamically by interaction

RMCP MIDI Receiver

Gestures

RMCP Player Server

Take the human player’s performance as input(computer player’s brain)

Understand and improvise musical performance

Leaning to the left or right

RMCP Beat Provider

Direct the player on the selected side to take up solo

Keep the tempo of the whole performance

Pointing to his head

RMCP Sound Server

Direct the other players to return to the theme

Output sounds of the computer players’ performances

Computer player frequently turns his eyes toanother player to watch for gestures
toward the end of each song part

RMCP Animation Server(computer player’s body)

Display each computer player through CG animation
RMCP Camera Analyzer

Computer player nods to the player who made gesture
No.12

1 Predictt-1 t t+1

Motions

theme

PianoSession Understanding

Me (Bass) Drums

FeaturesNo.16(computer player’s eyes)

Recognize the human player’s gestures through camera

6. Experiments and Results
Conditions
Four-beat jazz standard‘Take the "A" Train’

Theme, Chord progression, Key signature provided
Tempo: 187-230 M.M., constant

Results
Achieve a jazz session in which all players interacted
without the fixed leader-follower relationshipCG animation gave the pianist a greater feeling of
being at a live performance
Achieve multimodal interaction using sounds and gestures
No.17

CG output (bassist and drummer)

No.18

7. Conclusion
Summary
VirJa Session: Vir tual Ja zz Session System
All players listen to the other players
see each others’ bodies and gestures
Improvise without fixed leader-follower relationship
Interact using both musical sounds and gestures
Implemented on distributed workstations
No.19

Future Work
Upgrade the system
Follow tempo changes
Support other configurations:piano trio in which all players are computer players
sessions with different numbers of players

Remote jazz session
Our implementation facilitates remote jazz session

Freeware distribution (JAVA / VRML version)
http://www.info.waseda.ac.jp/muraoka/members/goto/PROJ/virja.html
No.20