Learning-Based Jam Session System for A Guitar Trio
Masatoshi Hamanaka∗ , Masataka Goto∗∗,∗∗∗ , and Nobuyuki Otsu∗∗∗,∗∗

University of Tsukuba, ∗∗ “Information and Human Activity,” PRESTO, JST
National Institute of Advanced Industrial Science and Technology (AIST)
m.hamanaka@aist.go.jp∗∗∗Abstract
This paper describes a jam session system that enables
a human player to interplay with virtual players, eachof which imitates musical reactions of a human player.
Previous session systems have parameters for altering
a way of reacting but have not been able to imitatesuch reactions. Our system can obtain the reaction
model of a human player—that is, the characteristic
way that player reacts to the other players—by learning the relationship between MIDI data of music theplayer listens to and MIDI data of music improvisedby the player. Experimental results show that the reaction model of any player participating in a guitar
trio session can be learned from the MIDI recording ofthat session.

1Introduction

This study aims to make a jam session system
in which virtual players react as if they were actual
human players with various characters. We want tomake it possible for human players to interact whenever they like with a virtual player imitating anyone
they want to perform with, a familiar, professional,
or deceased player... even themselves. What is mostimportant in imitating players is to acquire the reaction model of the target human player—that is, toacquire a model of the characteristic manner in whichthat target player reacts to the performances of otherplayers. The imitating virtual player can then improvise according to this reaction model.
Previous session systems have not been able to imitate a human player’s reaction. Some systems [1][2]concentrate on following the performance of a humansoloist without considering the individual character ofthe virtual player. Although JASPER [3] has a set ofrules that determine the system reactions and VirJa
Session [4] has parameters for altering a way of reacting, those systems cannot learn the reaction model ofan actual player.
Our jam session system makes it possible for the reaction model of a target human player to be acquired
from a MIDI recording of a session in which that player
participated. The system statistically learns the re-lationship between the MIDI data of the music thetarget player listens to and MIDI data of the music
improvised by that player. In other words, the system learns the relationship between the input and theoutput of the target player. The main advantage ofthis approach is that it is not necessary to examine
the target player directly: all we need to build themodel is a recording of a session in which that player
participated.

2

Learning-Based Session
SystemOur system deals with constant-tempo 12-bar blues
performed by a guitar trio consisting of a human guitarist and two virtual guitarists. The three playerstake the solo part one after another without a ﬁxed
leader-follower relationship. We chose the guitar trio
conﬁguration because we can use the performance ofany player in MIDI recordings when learning the reaction model.
The system has two session modes, a learning mode
and a session mode. In the learning mode (in Section 2.1), the system acquires reaction models in nonreal time. These models are stored in a database anddiﬀerent reaction models can be assigned to the two
virtual players before the session play. In the sessionmode (Section 2.2), a human player can interact withthe virtual players in real time. As shown in Figure 1,each virtual player listens to the performances of all
the players, including its own, and uses the reaction
model to determine what its next reaction (outputperformance) will be.
2.1

Learning ModeTo create a virtual player that reacts as the actual human player does, it is necessary to acquire theactual player’s individual reaction model. The main
issue in acquiring the reaction model is to learn therelationship between the input and the output of thetarget player in MIDI recordings. This can be formulated as a problem of obtaining the mapping fromthe input to the target player’s output. The direct

pastVirtualplayer A

Virtualplayer B

Humanplayer C

present
past performance of A

input performances of the three players

futurePlayer A’s
reaction model

Virtualplayer B

Virtualplayer A

next performance of A

Humanplayer C

low-level features of MIDI data
impression space

past performance of B next performance of B
Player B’s
reaction model

past performance of C

Player B’s reaction model
Player A’s reaction modelnext performance of C

intention space

Figure 1: Player’s reaction model.
Output performances

1) Impression space
The impression space represents the subjective impression derived from the MIDI input. By applying
principal component analysis (PCA) to the results ofsubjective evaluations of various MIDI performances,
we determined three coordinate axes of the impression space. PCA is a statistical method for reducing
the number of dimensions while capturing the major
variance in a large data set. While listening the performance, a subject evaluated 10 impression words subjectively by rating them on a scale of one to seven.
The three selected axes of the impression space represent qualities that can be described as appealing,energetic, and heavy. To obtain a vector in this space,
an impression vector corresponding to the MIDI input, we use canonical correlation analysis (CCA) .
This analysis maximizes the correlation between various low-level features of the MIDI input (such as pitch,counts of notes, tensions, and pitch bend) and the corresponding subjective evaluation. Since an impression
vector is obtained from each player’s performance,
we have at every moment three impression vectors(Figure 2) .
The impression space is necessary for learning therelationship between various input performances andthe corresponding output performances. If we represent the input performances as short MIDI segments
without using the impression space, the same MIDI
segments will not be repeated in a diﬀerent session.
The impression space enables to abstract subjective
impressions from input MIDI data and those impres-Figure 2: Player architecture.
sions can be repeated. Even if two segments of theinput MIDI data diﬀer, they can be represented as asimilar vector in the impression space as long as theygive the same impression.
Figure 3 shows the transition of the rated values
for the impression word ”appealing.” The black line
represents the value calculated by the system and thegray line represents the value evaluated by a humanlistener. For 92 percent of the performance, the calculated and subjectively evaluated values do not diﬀer
by more than 1.
2) Intention space
The intention space represents the intention of theplayer improvising the output. A vector in this space,an intention vector, determines the feeling of the nextoutput. It is used to select short MIDI phrases froma phrase database, and the output MIDI performanceis generated by connecting the selected phrases.
Without the intention space, it is diﬃcult to learnsystemsubject

765scale

MIDI-level learning of this mapping, however, is too
diﬃcult because the same MIDI-level situation rarely
occurs more than once and the mapping itself is toosparse. We therefore introduce two intermediate subjective spaces: an impression space and an intention
space (Figure 2) .

43212468101214161820

22 24[bars]

Figure 3: Transition of the rated values calculatedby the system and evaluated by a subject (for the impression word ”appealing”).

the relationship between impression vectors and output MIDI data because in actual MIDI recordings various output can occur when the input data gives thesame impression. The intention space makes it easierto learn the player’s reaction model.
The intention space is constructed by using multidimensional scaling (MDS) [5], such that intention
vectors are distributed with proximities proportional
to subjective similarities of short phrases corresponding to those vectors. The dimensions of this space
were determined from the MDS results to be three.
Because the number of the short phrases is limited, those phrases are sparsely placed in the intention space. When generating the output, the systemselects the output phrase close to the determined intention vector: an appropriate phrase can be selectedeven if the phrase database does not have a phrasethat is exactly placed on the intention vector.
3) Reaction model
We can regard the mapping from the impression
space to the intention space as the reaction model.
To learn this mapping function statistically, we obtained various training sets from the target session
recordings. These sets are pairs of impression vectors
obtained from the three players during a sequence of
past twelve bars and the corresponding next intention
vector. For this learning we use Gaussian radial basis
function (RBF) networks [6]. The RBF networks have
one hidden layer with nonlinear inputs, and each nodein the hidden layer computes the distance between theinput vector and the center of the corresponding radialbasis function. The RBF networks have good generalization ability and can learn a nonlinear mapping
function we are dealing with.
2.2

Session Mode

Using the reaction model acquired in the learningmode, each virtual player improvises while reacting to
the human player and the other virtual player. The
processing ﬂow of each virtual player can be summarized as follows:
1. The low-level features of MIDI performances of
all the players are calculated at every 1/48 bar.
2. Every 1/12 bar the three impression vectors areobtained from the low-level features.
3. At the beginning of every bar, the intention vector is determined by feeding the reaction model
past impression vectors.
4. The output performance is generated by connecting short phrases selected from a phrase

Virtual player's panel

Virtual player's panel

Human player's panel

Figure 4: Screen snapshot of the system output.
database that contains a hundred phrases,ﬁfty solo-style phrases, and ﬁfty backing-stylephrases. Each phrase is selected, according to
the determined intention vector, by considering
the ﬁtness for the chord progression. A virtual
player can start a solo performance at any bar.
Note that the reaction model can predict the next
intention vector from the impression vectors gathered
during the past twelve bars in real time: a virtualplayer thus does not get behind the other players.

3

Experimental Results

We have implemented the proposed system on a
personal computer (with a Pentium III 650 MHz processor) and Figure 4 shows a screen snapshot of thesystem output. In this ﬁgure there are three columns(called player panels) , each corresponding to a diﬀerent player. The toggle switch on the top of each panelin this ﬁgure indicates whether the panel is for a virtual player or a human player, and in each panel there
are two boxes representing three-dimensional spaces:
the upper box is the impression space and the lower
box is the intention space. The sphere in each box indicates the current value of the impression or intention
vector.
In our experiments, after recording a session performance of three human guitarists playing MIDI guitars,
we ﬁrst made the system learn the reaction model ofeach of them. We used a metronome sound to keep
the tempo (120 M.M.) when recording, and the total length of this recording session was 144 bars. We
then let ﬁve human guitarists use the system in thesession mode. The system indeed enabled each human1614121086

Virtual player A
Virtual player B
Human player

420012243648[bars]

Figure 5: Transition of intention vectors of three players. (This ﬁgure charts one component of intention
vectors that have three components.)
guitarist to interact with two virtual guitarists, eachwith a diﬀerent reaction model.
To ﬁnd out how well a virtual player could imitate
a human player, we asked a human player to performwith a virtual player A imitating him and a virtualplayer B imitating a diﬀerent player. The humanplayer and the virtual player imitating him tended totake the solo at almost the same time and to perform
phrases that felt similar. Figure 5 shows the transition of intention vectors of three players during 48 bars
where the intention vectors of the virtual player A andthe human player are particularly similar. Examining
all the values of the intention vectors during the session, we compared the distances between the intention
vectors of the virtual players and the human player.
Over 144 bars the average distance between the intention vectors of the human player and the virtual player
imitating him was signiﬁcantly smaller than that between the intention vectors of the human player andthe virtual player imitating a diﬀerent player. These
results showed that our system learned the reaction
model from the MIDI recordings of sessions.
Furthermore, ﬁve guitarists who performed with thesystem remarked that each virtual player performed
characteristically. In particular, the human player
who participated a jam session with a virtual playerthat imitated himself remarked that it was not comfortable to play with his virtual player because he felt
himself mimicked. We think that this means the virtual player’s RBF networks actually predicted the human player’s intention.

4Conclusion

We have described a session system in which a human guitarist and two virtual guitarists imitating human guitarists interact with each other. It is based on
the learning of a reaction model that is the mappingfrom the past performances of all the three players
to the next performance of the imitated player. The
experimental results showed that our system can imitate musical reactions of human players. We plan toextend the system so that it can be applied to othermusical instruments, such as piano and drums.

5Acknowledgments

We thank Daisuke Hashimoto, Shinichi Chida, andYasuhiro Saita for cooperating in the experiment asplayers.

References[1] M. Nishijima and K. Watanabe: Interactive music
composer on neural networks, Proc. of ICMC, pp.
53-56, 1992.
[2] Y. Aono, H. Katayose and S. Inokuchi: An improvisational accompaniment system observing performer’s musical gesture, Proc. of ICMC, pp. 106107, 1995.
[3] S. Wake, H. Kato, N. Saiwaki and S. Inokuchi: Cooperative Musical Partner System Using TensionParameter: JASPER (Jam Session Partner) ,Trans. of IPS Japan, Vol. 35, No. 7, pp. 1469-1481,
1994. (in Japanese)[4] M. Goto, I. Hidaka, H. Matsumoto, Y. Kuroda andY. Muraoka: A Jazz Session System for Interplay
among All Players — VirJa Session (Virtual JazzSession System) —, Proc. of ICMC, pp. 346-349,
1996.
[5] J.B. Kruskal and M. Wish, Multidimensional Scaling, Sage Publications, 1978.
[6] S.Chen, C.F.N. Cowan and P.M. Great: Orthogonal Least Sequares Learning Algorithm for Radial
Basis Function Networks, IEEE Transactions on
Neural Networks, Vol.4, pp.246-257, 1991.