指揮者のジェスチャー認識によるカオス音楽生成システム
Chaotic Music Generation System Based on Music Conductor
Gesture Recognition
Shuai Chen

Yoichiro Maeda

Yasutake Takahashi

Graduate School of Engineering, University of Fukui
Abstract: In the research of interactive chaotic sound, we propose a music generation method,that enables the computer to generate the music automatically, and then the music will be arranged
under the human music conductor’s gestures. In this research, the proposed method is designed
not only generating the music automatically by using network of chaotic elements in realtime but
also following the requirements of human. Music conductor gesture recognition and basic music
theories are embedded in the algorithm, so that the music will be richer. This system is expected toapply to the ﬁelds of low-cost background music generation, the interactive entertainment systems,
the education tools for conductors, and so on.

1

Introduction

There are already existing interactive systems controlling music or sound based on human body movements, for example, Sound Sculpting[1]. It proposed a
new approach to mapping hand movements to soundthrough shapes of a virtual input device controlled
by both hands. The attributes of the virtual object are translated into parameters for real-time soundediting within MAX/MSP. And a vision-based system, Body-Brush, has been proposed, that captures
the entire human body motions and gestures for 3D
painting synthesis and musical sound generation[2].
And also, instead of the virtual acquisition of body
movements, Cyber Composer, introduced as a music
generation system, that melody ﬂow and musical expressions can be controlled and generated by wearing
motion-sensing gloves[3].
In order to arouse the interest from everyone, it isnecessary to make the users control more in the way togenerate their music. In addition, in order to ensurethat the generated music makes every user feel easy
to understand and easy to control, music conductor
gestures must be applied in this paper.
In this research, the proposed method is designed,
not only following the requirements of people but also
generating the music automatically by using Interactive Chaotic Amusement System (ICAS)[4]. Music
conductor gestures and basic music theories are em-

bedded in the algorithm, so that, after the music havebeen generated, it will be arranged by the human’s
hand conducting gestures. The users use their hand
gestures to interact with the computer for generating
the music. Human intention and chaotic calculation
generate sounds with mutual multiplier eﬀects.

2

Proposed Music Generation
SystemWe propose a method to generate the music thatchanges according to the hand gesture of operator inreal-time, and not simply mapping the gestures directly into music. Through the application and extension of ICAS, we make it possible to combine theFig. 1: Overview of Music Generation Systemdiversity and the randomness of computer-generated
music with the human requirements for the music.
The overview of our system is shown in Fig. 1. Fig.
2 shows the algorithm of the proposed system.
In this research, ﬁrst we input the GCM parameters by using GUI with these data, and the pitch
and length of sound are generated by these parameters. And then, with the selected rhythm type in
the GUI, we adjust and accompanied the sound withthe rhythm and chord progression, and we achieve to
make our output sounds more like music. We capture
the operator’s hand gesture by Kinect camera, andﬁnd out the hand position from the input data, thenﬁgure out the center point of hand and wrist as thecharacteristic point of hand gesture. With the coordinates of those points of both hands in several serial
frames, we can calculate the amplitude, speed andacceleration of the motion of hand. Finally, we arranged the automatic generated music with the dataof recognized hand gesture.

where xi (t) shows the state of element i at a discrete
time t, f (x) shows logistic map, a parameter e shows
the strength of the entire combination, and N shows
the total number of elements.

3

3.2

Overview of ICAS

Network of chaotic elements has the connected elements of chaos which are coupled to mainly networklike. And these chaotic elements are given in the formof diﬀerential equation. It is also known as large-scale
coupled map, proposed by K.Kaneko[5]. By using thistheory, it is possible to reduce a complex and various
behavior of the entire map. In addition, the combination structure of network of chaotic elements can be
HumanOperation

SystemFlow Chart

Input GCM parameter,rhythm and tonality

Rhythm pattern andchord progression
determine

Chaotic soundgeneration

Automatic music
generation
Hand Gesture

Capture gesture from
Kinect Camera
Music arrangement

Listening Music

Output themusic
Satisfied ?

3.1

NO

YES
End

Fig. 2: Algorithm of Music Generation System

Globally Coupled Map

Globally Coupled Map(GCM) is a model of nonlinear system with a global connected chaotic network, that changed by all other elements interactingwith the same degree of intensity. The equation of
GCM is shown as follows:f (xj (t)) = 1 − ax2i (t)[i = 1, 2, ..., n],xi (t + 1) = [1 − e]f (xi (t)) +e f (xj (t)),
N j=1

(1)

N

(2)

About ICAS

In this research, we propose some further works
of Interactive Chaotic Amusement System (ICAS)[4].
By using ICAS which united the chaotic elements togenerate various sounds by GCM, favorite sounds foran operator appear as a whole adjusting the parameter of GCM. Its control is enabled by the humanoperator, and the sound is tuned by a visual information.

4

Start
Input Parameters
by using GUI

divided into Coupled Map Lattice (CML), and Globally Coupled Map (GCM). In this research, the Globally Coupled Map has been used in ICAS.

Music Generation

The ordinary ICAS system united the chaotic elements to generate sound by the Network of Chaotic
Elements, and its control is enable by the operator. In
our music generation system, we use ICAS to generate the basic sound according to the operator’s settingof tonality, and we proposed a method to select thechord progression and rhythm pattern automatically.
After this, we obtain the basic sound, generated by
ICAS, the chord progression and rhythm, generated
by our automatic selection method.
Next we synchronize these parts together, to makesure each part starts and ends at the same time. We
have four diﬀerent sound tracks, which all generated
individually, and we can let this four tracks play on
various sorts of instruments.

1Sound Pitch

Sound Length

GCM 1

GCM 2

Sound 1

Pitch Lh1

Length Lt1

Sound 2

Pitch Lh2

Length Lt2

3

Chord Progression

Constant

4

Pitch Lh3

LengthLtLt
Length33

Constant

Rhythm Pattern

Pitch Lh4

LengthLengthLt
Lt4 4Sound 3

2







 













 


















a) Tango Rhythm
Sound 4

1

2

Fig. 3: Sound Generation by ICAS

3

4

4.1

Sound Generation by ICAS

Fig. 3 shows the basic sound generation idea by using ICAS. Unlike the ordinary ICAS, which generates
the pitch, length and volume all by GCM, here we
only use the GCM to generate the pitch and lengthfor sound 1 and sound 2. The pitch of sound 3 andthe length of sound 4 are depend on the chord progression and rhythm pattern respectively. The soundlength of sound 3 is ﬁxed as a quarter note, and thepitch of sound 4 is also ﬁxed as the lower tonic of theselected tune.
The volume will be controlled by hand gesture directly. The generated sounds that from sound 1 tosound 4 are output simultaneously. While each noteis generated respectively, sound i (i=1,2) is produced
under the logistic map of Lhi , Lti . In addition, thepitch of Lh1 and Lh2 have been coupled within a
global map GCM1. Similarly, the length are coupled with the global map GCM2. The output ofsound generation xi (t) according to Eq. (2), is within−1.0 ≦ xi (t) < 1.0, and this range is divided as follows.
1. Tuning of pitch (GCM1)
We divided this range into 20, thus each small
range covers 0.1. These ranges corresponding to
the pitch in MIDI, the exact pitch of these 20
ranges will be changed if the selected tonalitychanges. When the tonality is C major, theseare from C−, one octave lower than center C, to
A+, two octaves higher.
2. Tuning of sound length (GCM2)
We deﬁne the longest length as a whole note,
which lasts 2000(msec), as a result, we divided
the range into 1:2:4:2:1, corresponding to the length

 















 



















b) Rumba Rhythm
1

2

3

4

  

  

  

  

 





























 



 







c) Samba Rhythm
Fig. 4: Typical Rhythm Patternsof 125:250:500:1000:2000(msec), as the quarter
note have the highest probability of occurrence.
Hence, if we take the quarter note as one beat,then these sounds will be played at the speed of
120 notes per minutes.

4.2

Rhythm Patterns

We take the rhythm into account, in our music generation system. Rhythmic phrases and patterns havebeen passing on diﬀerent approaches in diﬀerent music style throughout the world. Since the purpose of
us is to try to add the rhythm patterns into ICAS
generation, and intend to have a better output. In
order to conﬁrm our ideas, we come up with a simple
implementation of rhythm. In the present stage of thegeneration experiment, we only added three diﬀerent
rhythms: tango, rumba and samba. For each of theserhythm phrases, there are several typical patterns[6].
Fig. 4 shows the typical rhythm patterns of tango,
rumba and samba, which have been utilized in thisresearch.

4.3

Start Point

Chord Progression

The chord progression is actually a series of musical chords, this series of musical chords can create
the harmony. In the western classical notation, Roman numerals are used to number the chords built
on the scale. For example, I is indicated to the ﬁrst
note of a key, and a V in the key of C is indicatedto G chord, but in the key of A it indicated to E
chord. They usually occur on an accented beat, andwith the rhythm goes on, the chord shifts from one toanother. These chord shifts are undergoing some regular patterns, here we used the common three chords
progression: I - IV - V, for key of C, this chord progression turn out to be Chord C, Chord F and Chord
G. All the chord progression assembled by these threechords, will be always sounds harmony. We randomly
select one chord from the three chords, and make it
occurs at each accented beat of the rhythm.

Characteristic point

Hand
motionFig. 5: Hand Tracing Shape of 2/2 or 2/4 Time

5.1

Hand Gesture for Beat and Tempo

Synchronization is very important in our system, although we generate the four tracks in the same timescale, because any little delay or dislocate between two
tracks will cause the output sounds great disarray. After we recognize this problem in the experiments, we
tried to solve it by adjust the error of note length generation. The longest generated note lasts 2000(msec),
if we let the quarter note as one beat, and we can get
the smallest section with four beats in it. When every
section ﬁnished, we eliminate the error between four
tracks once. If the tonality or the rhythm have beenchanged, then all the tracks reset back into zero, andstart the generation from the beginning again.

The beat of the music, is usually expressed by the
conductor’s right hand. The shape of hand is waving in the air at every bar (a short section of music)
depending on the time signature of certain piece ofmusic, every change from the downward to upward ofhand movement is called one beat. Fig. 5 shows thesimplest common beat patterns considering the 2/2
or 2/4 time. At this stage, we used this pattern only.
The length of each beat, the time that the handmoves downward to upward once, determines the lengthof the note, and the speed of the music. Music conductor usually changes the wave speed of his/her hands
to transmit to musicians, the music here should beplayed in a diﬀerent speed.
Therefore, in our system, we calculate the the timeof hand downward to upward, and take the speed of
the characteristic point of hand as a characteristic
value of sound length. We change the tempo from60 Beats Per Minute (BPM) to 120 BPM, according
to the range of moving speed of hand characteristic
point from 300 pixels per second to 3000 pixels per
second.

5

5.2

4.4

Synchronization

Hand Gesture Recognition

Conducting is a means of communicating artistic
directions to performers during a performance. Although it is almost subjective and a variety of diﬀerent
conducting styles exists depending upon the training
and sophistication of the conductor, there are many
formal rules on how to conduct correctly. Despite awide variety of styles, a number of standard conventions have developed.

Hand Gesture for Volume

The music dynamics is indicated by the amplitude
of conductor’s hand movement, the larger shape expresses the louder sound. The conductors usually use
their left hand, while right hand have been used toshow the beat, to evince the extent of volume. Whenthe hand moves upward (usually palm-up) represents
a crescendo, and when the hand moves downward(usually palm-down) means a diminuendo. Changing
the amplitude of hand movement depending on thecircumstances, often leads the changing of the characteristic of the music. Here we try to make it simple,

Fig. 6: GUI of ICAS Simulator
so we use the amplitude of the hand as a characteristic value of volume. If the conductor wave his handsof a great range in a beat, the output music volume
will be great, and if the hand waving is smaller, output volume will be turned down. Here we mapped
the range of amplitude of hand characteristic pointfrom 20 pixels to 500 pixels, to the MIDI volume in
the range of 0 to 127.

6

Experiment

We have implemented the system described in thispaper to evaluate the eﬀect of the system. We designed two experiments, to ﬁnd out the diﬀerence ofthe output sound, when the input hand gesture havediﬀerence in speed and amplitude.

6.1

ICAS Simulator

The system we have simulated, is programmed under Microsoft Visual C++ and Cycling’74 company’s
graphical programming tool MAX/MSP. The GUI of
improved ICAS simulator is shown in Fig. 6. This
GUI mainly shows about the music generation, we
can set the sound tonality and length here. And before our conducting, we can listen to the automaticalgenerated music ﬁrst adjusting the parameters. When
we think it become interesting enough, we start our
conducting, and will let the music richer, more diverse.
The GUI of hand conducting gesture recognition isshown in Fig. 7. We get the operator’s music command gesture using a Microsoft Kinect camera. The
cross mark shows the center point of hand and wrist.
In this GUI, we calculate the hand speed and amplitude in realtime, and we can get hand direction data
easily, all these data will be send to the music generation and music arrangement program.

Fig. 7: GUI of Hand Conducting Gesture Recognition

6.2

Experimental Results

We performed an experiment aiming to verify thehand gesture and output sound associated well witheach other. First, we disconnect the hand conducting parts, the output music will be automatic generated and without arrangement. We record the output
music and get its sample music score shown in Fig.
8. Second, we connect the hand conducting part andlet the operator doing conducting gesture, and try tochange the tempo and volume of the output music.
Sample music score of second experiment is shown in
Fig. 9.
At ﬁrst, the operator conducts at a normal speed,
and then waves hand fast. In Fig. 8, the notes are stable and changes slowly. In Fig. 9, obviously diﬀerent
from Fig. 8, the generated notes are richer and changs
the tempo quite match with the operator’s hand conducting, in the ﬁrst four section, the generated musicat a normal speed, and then become fast from section
ﬁve to section seven.
As a result of the simulation, we conﬁrmed that thegenerated music has more interactivity than the simple ICAS system and we can get the desired soundroughly by changing our hand gestures, instead of adjusting the GCM parameters directly.
In this experiment, we try to ﬁnd out that whether
the music after arrangement by hand gesture can give
listener a better impression than the music generated
only by computer. In this research we use the Semantic Diﬀerential method to ﬁnd out how the users
think about our output music, we have prepared 10
pairs of bipolar adjectives to do this experiment, andwe consider these 10 pairs of items have the same importance. Ten people around twenty years old, act assubjects in this experiment. We focus on the diﬀerent
feelings of subjects when hearing the output music
which generated by ICAS only or generated by automatically rhythm and chord progression with ICAS

synchronization, or generated by include all above andadd the hand conducing arrange the music in realtime. Hence, we have to avoid the interference causedby other variables. In this experiment, we set all theother variables as a constant. From the evaluation
result shown in Fig. 10, we can ﬁnd that the output
music of hand conducted have enriched the generated
music to more novel, impressive, complex, intense andfantastic than the former ones.

7[6] A. Blatter: Revisiting music theory: a guide to
the practice, p.28, 2007.
 



5



In this paper, we proposed the music conductor gesture arranged music generation system. We generated
the chaotic sound ﬁrst by ICAS, the conducting hand
gestures have been used to arranging the output music. We took the rhythm and chord progression ofmusic into consideration. On the other hand, the conducting gesture was also considered the simplest situation of 2/2 or 2/4 time. For the further research, wewill enrich the musical expressions and the conducting gestures to achieve a better output sound and ahigher level of interaction.





 



 

 

    



 




 





 


       
 


  



 










                   






 






         



References[3] H. S. Ip, C. K. Law, B. Kwong: Cyber Composer:
Hand Gesture-Driven Intelligent Music Composition and Generation, Proceedings of the 11th
International Multimedia Modelling Conference(MMM ’05), Vol. 1550, pp. 5502-5505, 2005.









   





  
 
  
        






 

  

  



 
 

 
 

   








 
 

 










     




 

  







  


 





     




 


 



 



 





     













   
 







            
 






Fig. 9: Score of Output Music with Gesture Arrangement (C major, Tango, Another Piece of Output Music)

Negative

Positive[4] M. Yamamoto, Y. Maeda: Interactive Kansei
System for Sound Generation Based on Networkof Chaotic Elements, Proc. of the Fourth International Symposium on Human and Artificial Intelligence Systems: From Control to Autonomy(HART 2004), pp. 315-320, 2004.
[5] K. Kaneko: Partition complexity in a network ofchaotic elements, Jounal of Physics A, pp. 21072119, 1991.



Fig. 8: Score of Output Music without Gesture Arrangement (C major, Tango) [2] H. S. Ip, H. Young, C. C. Tang: Body-Brush: A
Bodydriven Interface for Visual Aesthetics, ACM
International Conference on Multimedia 2002,
pp. 664-665, 2002.



 

5

[1] A. Mulder, S. Fels: Sound Sculpting: Manipulating Sound through Virtual Sculpting, Western
Computer Graphics Symposium, pp. 15-23, 1998.





 


    

 


 

Conclusions

Fig. 10: Kansei Evaluation Results