A Jazz Session System for Interplay among All Players
VirJa Session (Virtual Jazz Session System)
Masataka Goto Isao Hidaka Hideaki Matsumoto
Yosuke Kuroda Yoichi Muraoka
School of Science and Engineering, Waseda University
3-4-1 Ohkubo Shinjuku-ku, Tokyo 169, JAPAN.
fgoto, hidaka, matumoto, ykuroda, muraokag@muraoka.info.waseda.ac.jp

Abstract This paper presents a jazz session system in which each player is independent and caninterplay with all other players. In most previous systems, computer players only reacted to humanplayer’s performance with a fixed leader-follower relationship. Our system enables computer players
to listen to other computer players’ performances as well as human’s performance and to interact witheach other. Moreover, all players can communicate not only by listening to other players’ performances
but also by seeing each others’ bodies and gestures. In our current implementation, the system deals
with a jazz piano trio consisting of a human pianist, a computer bassist, and a computer drummer. Thesecomputer players have been implemented as separate processes on several distributed workstations.

1 Introduction

The human player sees the bodies and gestures of othercomputer players shown on computer graphics (CG),and thus can feel their presence as if they were actually playing together. In addition, each computer player
sees the gestures of the human player through a video
camera, and recognizes the virtual gestures of the othercomputer player through a computer network. Thus, allplayers can interact with each other using both auditory
and visual information.
In our current implementation, our system deals witha four-beat jazz standard with a constant tempo performed by a piano trio consisting of a human pianist, a
computer bassist, and a computer drummer. To perform
this computationally intensive task in real time, the system has been implemented on several distributed workstations connected to a computer network. The computer bassist and drummer are implemented as separate
processes to facilitate system implementation and expansion. Each of them generates and sends MIDI dataof his performance reacting to MIDI data received fromboth the human pianist and the other computer player.
In our experiment, we achieved a jazz session in which
all players improvised and interacted using both musical sounds and gestures.

It is important in jazz that all players improvise together
while reacting to other players’ performances with noleader-follower relationship. This is called interplay,
and all players communicate by musical sounds and additional visual information such as gestures. The purpose of this research is to simulate the actual interactionthat occurs among human players on a computer systemthat enables interplay among humans and computers.
Most previous jazz session systems [Nishijima andWatanabe, 1992; Kondo et al., 1993; Wake et al., 1994] andautomatic accompaniment systems [Dannenberg, 1984;
Vercoe, 1984; Baird et al., 1989; Rowe, 1993; Horiuchi andTanaka, 1993; Hidaka et al., 1995] generated several ac-companiment parts together, which only reacted tothe soloist’s (the human player’s) performance. There
was a fixed relationship where the human player was
the leader and the other computer players were thefollowers. Since computer players only listened tothe soloist’s performance, they were not able to interact with each other. Although one previous paper [Kanamori et al., 1993] mentioned interaction among
computer players, it did not discuss the generation ofmusical performance and only focused on the musiclistening process.
Our jazz session system enables a human player andcomputer players to interact with each other without
a fixed leader-follower relationship. In our system, acomputer player not only reacts to the human player’s
performance but also plays even a kind of solo. Computer players can react to each other by listening to othercomputer players’ performances just as a human player
listens to all other players in an actual jazz session.
This paper moreover presents an advanced jazz session system called VirJa Session (Virtual Jazz Session System). VirJa Session enables all players tocommunicate not only by listening to other players’ performances but also by seeing each others’ bodies andgestures that indicate repetition or the end of a song.
Goto et al.

2

Jazz Session Model for Interplay
among All Players

Figure 1 shows a jazz session model in which all players listen and react to all other players. Although thefigure only depicts a piano trio configuration consistingof a human pianist, a computer bassist, and a computer
drummer, this model is applicable to various sessions
consisting of several players.
In the model, either human or computer can be selected for each player; only system implementation restricts the configuration. We can imagine sessions inwhich all players are human players and we can imagine sessions in which all players are computer players.
The former are useful for remote sessions via computernetworks. The latter are effective when various design346

ICMC Proceedings 1996

Player 1: Pianist

Player 1: Pianist
Human
Computer

Human
ComputerInteraction

Virtual Jazz Trio
Human
ComputerCa

mera

Computer

al

Human

Interaction

Human
Computer

Virtual Camera

ComputerFigure 1: Jazz session model for piano trio.

Real-time CG

Virtual Camera

Real-time CG

Figure 2: VirJa Session.

ers implement different computer players with various
characteristics. These computer players may be thoughtof as the designers’ substitutes, and these substitutes
can interact with each other.
We built a jazz session system for the piano trio as
illustrated in Figure 1. Both player 2 and 3 are computer players, and it is important that these computer
players are executed as separate processes; the systemshould not determine their performances together according only to the performance of player 1, the humanplayer. Computer player 2, for example, should listen
and react the performance of player 3 as well as the performance of player 1. Player 2 (bassist) and player 3(drummer) thus interact with each other through theirmusical performances.

ables the players to cooperate well through informationthat is difficult to exchange only using musical sounds.

3.1

Visualization of Computer Players on
CG animation

The entire body of each computer player and his musical instruments (bass or drums) are shown on a CGscreen. The CG player makes the following motions:(1) playing musical instruments according to his musical sounds, (2) keeping time to musical beats by foottapping or rocking his body, (3) making two kinds ofgestures described in Section 3.2, (4) nodding to showthat he understood a gesture, and (5) turning his eyes toanother player.
Visualization of a musician on CG was reported in[Katayose et al., 1993; Kamei et al., 1995], but these papers only focused on hand movements of a virtual guitarist. Although an earlier paper [Goto and Hashimoto,1993] discussed visualization of musicians, the system
only generated a CG dancer, not CG musicians.

3 Virtual Jazz Session System with CGand Camera 0 VirJa Session
Figure 2 shows the advanced jazz session system, VirJa
Session, in which all players communicate both bylistening to other players’ performances and by seeing
each others’ bodies and gestures. Each player presented
in the last section has only his ears to listen to the performances and only his hands and feet to play the musicalinstruments. Each player in the VirJa Session is additionally able to use his eyes to see other players and his
body to show his gestures. Since each player thus exchanges gestures in addition to musical sounds, we canachieve multimodal interaction among all players using
both auditory and visual information. This enables theplayers to feel the presence of other players as if they
were actually playing together, and enables audiences
to feel the presence of all players as if they were attending a live concert, and not listening to a compact disc.
The human player sees the bodies and gestures of thecomputer bassist and drummer shown through threedimensional CG animation in real time. Each computer player recognizes gestures of the human player
through a real video camera, and receives gestures ofthe other computer player through a virtual camera thatmeans passing messages corresponding to the gestures
via computer network. This exchange of gestures enICMC Proceedings 1996

Player 3: Drummerrame

HumanCa

Player 2: Bassist

Player 3: Drummer

alRe

Player 2: Bassist

Virtual Jazz TrioReInteraction

3.2

Multimodal Interaction

In VirJa Session, we introduce a song form called scenario, which allows a player to change his way of playing according to where he plays in it. A scenario isgiven in advance of a performance as a combination ofsong parts such as theme, player solos, and four verse1 .
The number of repetitions of each song part in the scenario is not pre-arranged, and is determined dynamically by interaction using gestures among players, as in
actual sessions among human players. Our current implementation only limits the maximum number of repetitions.
We introduce two gestures, one indicating that thenext player should begin his solo and one indicating thatthe players should return to the theme (reprise). The
first gesture is done by a player leaning to the left or1
Players, for example, may perform according to the following scenario: theme ) piano solo ) bass solo ) fourverse (piano solo and drums solo alternate every four measures) ) theme (reprise).

347

Goto et al.

right, and directs the player on the selected side2 to takeup the solo. The second gesture is done by a player
pointing to his head.
Toward the end of each song part in a scenario, eachcomputer player frequently turns his eyes (which show
where his attention is directed) to another player towatch for gestures indicating whether the current songpart should be repeated or the next part should be takenup. The human player should make a gesture while
computer players are looking at him. When a computer
player understands a gesture, he nods to the player who
made the gesture.

VirJa Session

Player 1: Pianist
Human
Computer

RMCP MIDI Receiver

RMCP Sound Server

M
RMCP Camera Analyzer

PianistG
Ethernet
M
Player 3: Drummer
M
G M
M GB
Human
Human
B G
B
B GA
ComputerComputer
A
M
M
RMCP Player Server
RMCP Player Server
B
B
A
A

Player 2: Bassist

RMCP Beat Provider

4 Computer Bassist and Drummer

RMCP Animation Server

Each computer player performs in three stages: Music Listening, Session Understanding, and PerformanceImprovising. In the Music Listening stage, he analyzes
the musical performance of each of the other players byusing the intention parameters described in [Hidaka etal., 1995] according to the character of musical instruments, and estimates how much each player leads thesession in the whole ensemble (leadership percentage).
In the Session Understanding stage, he considers thewhole musical relationships among players by predicting the next leadership percentage for each player basedon recent transitions of the leadership percentage. He
then determines how much he tries to lead the session
based on where he plays in a scenario; the bassist, forexample, tries to take a stronger leadership role during
the bass solo. In the Performance Improvising stage,
he chooses a pitch pattern, a loudness pattern, and arhythm pattern from pre-registered pattern databases.
These patterns are chosen in such a way that he can takethe leadership percentage determined in the Session Understanding stage. He then combines these patterns on
every beat to generate his musical performance.

BassistM Music Information
G Gesture Information

RMCP Beat Provider

RMCP Animation ServerDrummer
B Beat Information
A Animation Information

Figure 3: System configuration of VirJa Session on RMCP.

the basis of the server-client model. This implementation makes it possible to allocate these tasks on several distributed computers on the network, and facilitates system implementation and expansion.
In our current implementation the system works onthree workstations (SGI Indigo2 Impact, Extreme) connected to the Ethernet. We use MIDI (Musical Instrument Digital Interface) to handle the players’ performances and RMCP (Remote Music Control Protocol)[Goto and Hashimoto, 1993] for inter-process communication.

5.1

RMCPRMCP is a communication protocol on the UDP/IP
between servers and clients in a distributed cooperative system that integrates MIDI and LAN. RMCP
was designed to transmit symbolized musical information through networks. In VirJa Session, the following four kinds of information are broadcast as RMCP
packets: (1) music information (MIDI messages), (2)beat information (temporal positions of quarter notes,tempo), (3) gesture information (leaning to the left or
right, pointing to his head), and (4) animation information (direct a computer player on CG to make a gesture,
to nod, and to turn his eyes). This broadcast enables
several computers on the Ethernet to utilize and sharethe information in various ways at the same time. Since
each RMCP packet has a time stamp (msec-resolution)in its header, packets that are received by RMCP servers
before their time stamps can be processed on time andin order.

5 Implementation in Distributed
Computing Environment
The following six tasks are necessary in VirJa Session:
1. understand and improvise musical performancefor each computer player (computer player’sbrain)
2. keep the tempo of the whole performance
3. output sounds of the computer players’ performances for the human player
4. take the human player’s performance as input forthe computer players
5. display each computer player through CG animation (computer player’s body)
6. recognize the human player’s gestures through avideo camera (computer player’s eyes)
To perform these computationally intensive tasks in realtime, each of these tasks is implemented as a separate
software process, and these processes communicate on

5.2

Implementation of VirJa Session

Figure 3 shows the system configuration of VirJa Session on RMCP. The above-mentioned six tasks are executed as follows: task 1 (music understanding and improvising) is executed by two RMCP Player Servers forboth computer players, task 2 (tempo keep) is executedby the RMCP Beat Providers, task 3 (sound output) is2

The triangle arrangement of the three players facing eachother is given in advance.

Goto et al.

M

348

ICMC Proceedings 1996

have also presented our implementation on distributed
workstations using the RMCP, which facilitated system
implementation and expansion.
We plan to upgrade the system so that it can follow tempo changes and to support other configurations,such as a piano trio in which all players are computer
players, and sessions with different numbers of players. Future work will include the study of a remote jazzsession, since our implementation facilitates use of thesystem in settings where the players are not in the same
physical location.

Acknowledgments

Figure 4: An example of CG output (bassist and drummer).

We thank Mineko Ogata, Kaoru Asatani, Tetsuya Ohmori,and Shin’ichiro Mayuzumi for their cooperation in our experiments and their helpful comments.

executed by the RMCP Sound Server, task 4 (performance input) is executed by the RMCP MIDI Receiver,task 5 (CG output) is executed by the RMCP Animation
Servers, and task 6 (gesture recognition) is executed by
the RMCP Camera Analyzer.
The kinds of RMCP packets that these processes
broadcast or receive are also shown in Figure 3. EachRMCP Player Server, for example, receives music information broadcast by the RMCP MIDI Receiver andthe other RMCP Player Server, and broadcasts music
information of his improvised performance, which isused by the corresponding RMCP Animation Server togenerate CG motions.

References[Baird et al., 1989] Bridget Baird, Donald Blevins, and NoelZahler. The artificially intelligent computer performer on
the Macintosh II and a pattern matching algorithm for realtime interactive performance. In Proc. of ICMC 1989,
pages 13016, 1989.
[Dannenberg, 1984] Roger B. Dannenberg. An on-line algorithm for real-time accompaniment. In Proc. of ICMC
1984, pages 1930198, 1984.
[Goto and Hashimoto, 1993] Masataka Goto and Yuji Hashimoto. A distributed cooperative system to play MIDI instruments 0 toward a remote session 0 (in Japanese).
IPSJ SIG Notes, 93(109):108, 1993.
[Hidaka et al., 1995] Isao Hidaka, Masataka Goto, and Yoichi Muraoka. An automatic jazz accompaniment system
reacting to solo. In Proc. of ICMC 1995, pages 1670170,1995.
[Horiuchi and Tanaka, 1993] Yasuo Horiuchi and Hozumi
Tanaka. A computer accompaniment system with independence. In Proc. of ICMC 1993, pages 4180420, 1993.
[Kamei et al., 1995] K. Kamei, K. Sato, H. Katayose, and S.
Inokuchi. Animation of human motion by editing real images and employing 3-D hand model (in Japanese). Trans.
of IPS Japan, 36(2):3740382, 1995.
[Kanamori et al., 1993] T. Kanamori, H. Katayose, S. Inokuchi, H. Hirai, and Y. Niimi. Interpretation of musicality
in jazz improvisation using multi-agent model. In Proc. ofIAKTA/LIST Intl. Workshop on Knowledge Technology inthe Arts, pages 1070114, 1993.
[Katayose et al., 1993] H. Katayose, T. Kanamori, K. Kamei, Y. Nagashima, K. Sato, S. Inokuchi, and S. Simura. Virtual performer. In Proc. of ICMC 1993, pages1380145, 1993.
[Kondo et al., 1993] Kinya Kondo, Haruhiro Katayose, andSeiji Inokuchi. The session system reacting to the player’s
intention (in Japanese). In Proc. of the 46th Annual Convention IPS Japan, pages 1:37301:374, 1993.
[Nishijima and Watanabe, 1992] Masako Nishijima and Kazuyuki Watanabe. Interactive music composer based onneural networks. In Proc. of ICMC 1992, pages 53056,1992.
[Rowe, 1993] Robert Rowe. Interactive Music Systems. The
MIT Press, 1993.
[Vercoe, 1984] Barry Vercoe. The synthetic performer in thecontext of live performance. In Proc. of ICMC 1984, pages
1990200, 1984.
[Wake et al., 1994] S. Wake, H. Kato, N. Saiwaki, and S.
Inokuchi. Cooperative musical partner system using
tension parameter: JASPER (jam session partner) (inJapanese). Trans. of IPS Japan, 35(7):146901481, 1994.

6 Experiments and Results
The VirJa Session was tested on a four-beat jazz standard named ‘Take the “A” Train’ by a pianist who has
been playing jazz piano for 5 years (and classical pianofor 21 years). We provided the theme, chord progression, key signature, and tempo (187-230 M.M.) for thesystem in advance, and tried several scenarios that included theme, bass solo, piano solo, and four verse (piano and drums solo). A MIDI synthesizer (Korg 01/WFD) was used for the sound output and for the keyboard
on which the pianist played.
In our experiment, we achieved a jazz session inwhich all players interacted with each other without thefixed leader-follower relationship. The computer players as well as the pianist led the session and played solos by improvising. The interaction among all players
made the entire performance expressive and interesting.
As compared with sessions using only musical sounds,
the CG animation of the computer players (Figure 4shows an example of the CG output) gave the pianist agreater feeling of being at a live performance. We alsoachieved multimodal interaction among players using
the gestures in addition to musical sounds.

7 Conclusion
We have described a jazz session system called VirJa
Session in which all players listen to the other players’ performances and see each others’ bodies and gestures. In this system, the players were able to improvise without a fixed leader-follower relationship and tointeract using both musical sounds and gestures. We
ICMC Proceedings 1996

349

Goto et al.