PB1-08

The 1st International Conference on Advanced Imaging

A study of automatic page turning of musical scoresby detecting player's nods

Haruka JIBIKI*1, Toshikazu SHIMIZU*1, Yasuyuki SAITO*1,Eita NAKAMURA*2, Shigeki SAGAYAMA*2*1 Dept. of Information Engineering, National Institute of Technology, Kisarazu College*2 School of Interdisciplinary Mathematical Sciences, Dept. of Frontier Media Science, Meiji
University
In this study, we construct a page turning system for musical scores in PDF files on a computer. Pianoplayers often ask a page turner to turn over pages of a musical score, especially in case of that, the musicpiece is complex or its designated tempo is fast. However, it is necessary to turn pages by oneself during
practice. Turning pages by oneself is problematic with incomplete turnings and misses, because it should
be done very hurriedly. Piano players use both hands and feet. Thus, the only body part which can be used
is head, and in fact they usually indicate a turning to the page turner by nods. We pay attention to thisaction and develop software to detect it by using USB cameras. Moreover, we solve the face detectionproblem by using a hidden Markov model.
1

1. Introduction
In general, an instrument player turns over musicalscores by oneself. In contrast, a piano player asks a pageturner to turn over musical scores. However, it isnecessary to turn the pages by oneself during practice,even if the player plays long and complex music withboth hands at the same time without a break.
A German company Forma Bene developed an
automatic page turning device1) called “Volta Bene”(Fig.1). The device has an arm with a magnet and a footswitch and works on battery. The user puts iron chips on
the musical score in advance, and stamps the foot switch
to turn over a page. The arm swings like a car wiper, andthe magnet attracts the iron chip. Then the musical scoreis turned over. Although it has an advantage that the user
can use real musical scores during the practice, thedevice has many problems: (1) it often fails to turn over,(2) external power supply is necessary for stableperformance, (3) the player can not stamp the foot switch
when he performs the piano using pedal with both feet atthe same time, and so on.
Actually, piano players sign a human page turner bynods. Therefore, we focus on the nods to construct anautomatic page turning system for the musical score.
2. A method of turning over musical scores
2.1 Face tracking
There are some devices to detect player’s nods; forexample, USB camera, RGB-D camera, acceleration
sensor. Lately, it has been possible to produce RGB-D*1 2-11-1 Kiyomidai-higashi, Kisarazu, Chiba, Japane-mail: saito@j.kisarazu.ac.jp

camera and acceleration sensor at relatively low cost, andthey are commercially available at affordable prices.
However, in general, they have not been yet integrated inlaptops and mobile devices such as tablet terminal,smart-phone, and so on. Moreover, in the case of using
the acceleration sensor, the user must put it on the head,
which is not natural. On the other hand, USB camera iscommon now and equipped with standard laptops andmobile devices. Therefore, we use it in this study.
We have designed a face tracking system consisting oftwo stages. In the first stage, the system detects faceregion by using Haar-like characteristics in the whole
image obtained by the USB camera.
Next, the system extracts the central region of theinput image as a nose region. In comparison, the nose
region does not deform significantly. Moreover, we
assume that the piano player is in front of the piano, andthe distance between the player and the piano is limited
within a certain range. Thus the size of the apparent nose
does not change significantly. After this, to realize thehigh speed nose detection, the system focuses on aperiphery of the nose region instead of the whole image.

− 272 −

The 1st International Conference on Advanced ImagingcDescription
Number of conditions

nNumber of time series

sj(t)
(t)

State at certain time t ( j=1,2,…, c)
OutputInitial selected probability of state i(i = 1, 2, …,c)Transition probability from state i to jProbability that output o(t) is observed fromthe state sj(t).
Maximum probability on state j at t

Symbolori

2.2 Player’s nod estimation
To avoid effects of noises and player's head motions
which are not subject to detect, we introduce a hidden
Markov model (HMM) as follows.
We divide a player’s nod into following sections: (1)stationary, (2) moving down, (3) stationary, (4) movingup, (5) stationary. We regard each section as a “state” andconsider that the time series of the nod actions are output(see below) while transiting between the states. In thiscase, although only the outputs can be observed, it isdifficult to know directly whether they are output fromwhich state. Thus, the state transition is hidden.
Therefore, we introduce HMM which is known that it issuitable to model such time series. Suppose the first andlast “stationary” states are same, we design an HMM as
shown in Fig.2. Each state sj corresponds to each sectionof nod action: “stationary” (s1), “moving down” (s2),“stationary” (s3; may be skipped), and “moving up” (s4).
Each state transits to itself or next state stochastically.
Consider that each state outputs a distance dp whichindicates a difference of between current and previous
nose position of vertical direction as a nod action.
Moreover, we also consider distinguishing a player’s nodfrom just looking down, for example, the player looks at
the key of the piano. This problem is solved by counting
up the time length of the state transition which stays on
s3. However, when a predetermined time elapses while
the nose stays almost the same position (namely, staying
s1 or s3), all the values of dp in the buffer become close tozero because the buffer which stores the nose positions
consists of a finite length. Thus, it is difficult todistinguish whether the transition state is s1 or s3 only
using the distance dp. Therefore, we introduce one other
output da from the states which indicates the distance ofbetween the current and the average nose position ofvertical direction. If the nose stays at lower position, dabecomes a large value compared to staying at the initialposition. And then, the values used for the update
calculation of the average are vertical coordinates of thenose positions which are obtained only in time in which
the estimated transition state is s1.

aijb(sj(t),o(t))
pj(t)
qj(t)
z(t)

Storage for representation of optimum stateseries
State number of optimum series

Although we are not able to know the past transitedstate clearly, Viterbi algorithm is well known as anefficient solution of state transition estimation of theHMM. The detail of Viterbi algorithm is shown insection 3. At first, we determine the HMM paths ofplayer’s nods. If a player performs the nod, the order ofthe estimated state transition should be s1 s2 s4 s1. Or, inthe case of the order s1 s2 s3 s4 s1, the staying time of s3 isshort. In fact, because the HMM path includes self
transitions, the staying time length at each state isvarious. Hence the path matching is done after the timecompression as reducing some unchanged states, forexample, the time series s1 s2 s2 s2 s3 s3 s4 s4 s4 s1 iscompressed into s1 s2 s3 s4 s1. When time series of both dp
and da from analysis of the USB camera image are given
and the estimation results of Viterbi algorithm agree withabove paths, the page turning sign is issued.
3. Viterbi algorithm
The Viterbi algorithm can estimate the optimum pathof state transition of time series2). It is based on dynamicprogramming. Mathematical symbols which are used inthe Viterbi algorithm are shown in Table 1.
The Viterbi algorithm consists of four steps as follows:
Step 1 Initializationpi(1) = ri b(si(1), o(1))qi(1) = 0(i = 1, 2, …, c)
Step 2 Recursive calculation
pj(t) = maxi { pj(t-1) aij } b(sj(t), o(t))qj(t) = argmaxi { pi(t-1) aij}(t = 2, 3, …, n) (i, j = 1, 2, …, c)
Step 3 Terminationz(n) = argmaxi { pi(n) } (i = 1, 2, …, c)− 273 −

(1)
(2)

(3)(4)(5)

The 1st International Conference on Advanced Imaging

Step 4 Representation of state seriesz(t) = qz(t+1)(t+1) (t = n-1, …, 2, 1)(6)Note that the order of the estimating is reverse for t.
In case that all ri are set to non-zero probabilities, allstates are possible to become initial state. On the other
hand, if some of ri are given zero arbitrarily, the specifiedstates are only able to be initial states. Thus, it is possible
to control the path of the state transition to some extent.
4. Implementation
We implement the system by using Java3). Java isdesigned based on the concept of object-orientedprogramming, and to reduce the complexity of softwaredevelopment and maintenance, to increase thedevelopment efficiency and maintainability. Java is
excellent in versatility, and it is possible to perform thedevelopment and deployment of application software
platform-independent. In this study, although we develop
the system on a laptop, we will apply it on mobiledevices.
We use OpenCV4) for Java which is developed andreleased by Intel. It is a computer vision library of open
source. Currently, the development of the OpenCV has
been taken over by Willow Garage. The OpenCV is very
useful, and can be easily used. It has many methods, andthey work very quickly.
We introduce PDFRenderer5) which is a presentation
library for PDF file. A method RePaintDisplay()expresses a page of a PDF file. The argument of themethod is a page number. Suppose a class variable
nowPageNums indicate page number, we can use thefollowing method to display the next page:
RePaintDisplay(++nowPageNum);Thus, when the nod estimation system detects player’s
nods, above method with the argument is called.
5. Experimental results
We used a laptop whose CPU is Intel Core i5 1.6 GHz

and memory is 8 GB, and PDF musical scores which are
not processed such as embedding special symbols.
First, we defined initial probabilities ri, state transition
probabilities aij and output probabilities b(sj(t),o(t)) as
follows:
ri = 0.5 0.0 0.5 0.0 ,(7)§ 0 .8¨¨ 0.0
aij = ¨0.0
¨
¨ 0.2
©

0 .2 0 .0 0 .0 ·¸0 .6 0 .2 0 .2 ¸,0 .0 0 . 8 0 .2 ¸¸0.0 0.0 0.8 ¸¹(8)b(sj, o(t)) = b(sj, dp) b(sj, da),b(sj, dp) =b(sj, da) =12√2𝜋𝜎𝑝,𝑗12√2𝜋𝜎𝑎,𝑗exp (−exp (−(9)
2(𝑑𝑝 −𝜇𝑝,𝑗 )22𝜎𝑝,𝑗(𝑑𝑎 −𝜇𝑎 ,𝑗)2
22𝜎𝑎,𝑗),(10)),(11)

where dp is the distance between current and previous
nose position, da is the distance between current andaverage nose position. And we assumed parameters μand σ as follows:
μp = {0.0, 10.0, 0.0, -10.0},(12)2σp,j = {1.5, 2.0, 1.5, 2.0},(13)μa = {0.0, 15.0, 30.0, 15.0},(14)
σ2a,j = {1.5, 2.0, 1.5, 2.0},(15)
thus, b(sj(t), dp) and b(sj(t), da) are shown in Fig.4 andFig.5.
As the result of experiments for a professional piano
player, the system turned over all the pages accurately.
Fig.6 shows a result of the player’s nose position. Notethat the axes values are set according to image
coordinates. From this figure, it seems that when a lineof sight moves from top to bottom of musical score, thenose turns downward a little bit slowly. If the player
would like to turn over the page, the face turns down
quickly and greatly. On the other hand, in the case oflooking down simply, for example, the player looks at
the keyboard, the nose position stays in the bottom.
Fig.7 shows the other result which includes justlooking down actions. In this figure, the time sections
between 13 and 29, 41 and 64, 89 and 107 indicate their− 274 −

The 1st International Conference on Advanced Imagingactions. We confirmed that the page turning did not occur
during these time sections. Its process is valid. Inaddition, the average nose position does not change inthese time sections. On the other hand, the time sections
between 64 and 74, 113 and 118 demonstrate player’s
nods. In these cases, the page turning occurred properly.
6. Discussion
6.1 Aggressive use of horizontal head motions
In this study, horizontal head motions (clockwise andcounterclockwise rotations with respect to the verticalaxis of the head) are ignored because they do notcontribute to nods. If it is allowed to use some special
head motions, some functions are able to be realized. For
example, we can relate horizontal head motions toturning to the previous pages, and to jumping to the coda
page. Although this method is useful, such head motions
are not usually used for real piano performance.
Therefore, it is necessary to verify the effectiveness byusing this method in practice.
6.2 Combination of our method and fully automatic
page turning system
Our system can automatically turn over the pages.
However, it can be also said to be a kind of manual
operation software because the user instructs it by nods.
On the other hand, theoretically, fully automatic pageturning system will be realized if player’s performance
position in the music is recognized accurately. In thiscase, score following method6,7) is very important.
By using such system, the player can concentrate upon
the performance. However, the user sometimes wants toturn over the page before the playing position reaches thelast bar of the page, for example, the user has alreadymemorized the last measure of the page, or the beginningmeasure of the next page has complex passages. In suchcase, our system provides the page turning at any time.
Therefore a combination of our system and fully

automatic page turning system will bring about a more
flexible page turning system.
7. Conclusions
This paper presented a system of automatic pageturning of musical scores by detecting player's nods. We
implemented the system by OpenCV on Java using amethod of efficient face tracking. The nod estimation
was based on hidden Markov model, and the system
worked properly in real time.
As the future work, we will implement extensions
such as turning to a previous page, coping with repeat
with page jumping (da capo, Dal Segno, et cetera),
introducing a fully automatic page turning system, andso on. Moreover, we have planned implementing thissystem in mobile devices like smartphones and tablets.
Then, we will examine the ability of the system by alarge number of subjects in real music performance.
Acknowledgement: This study is supported in part by
Grant-in-Aid for Scientific Research from Japan Society
for the Promotion of Science, No. 26240025.
References
1)
2)
3)
4)
5)
6)

7)

− 275 −

Automatic page turning system (in Japanese),
http://inforent.dreamblog.jp/blog/78.html
Christopher M. Bishop: Pattern Recognition and Machine
Learning, pp.610-634 (Springer-Verlag, New York, 2006)
Java, https://www.java.com/en/
OpenCV, http://opencv.org/
PDFRenderer, https://developer.android.com/reference/
android/graphics/pdf/PdfRenderer.html
A. Arzt, G. Widmer, and S. Dixon: Automatic page turning
for musicians via real-time machine listening, Proc. ofECAI2008, pp.241-245 (2008)
E. Nakamura, T. Nakamura, Y. Saito, N. Ono, and S.
Sagayama: Outer-Product Hidden Markov Model andPolyphonic MIDI Score Following, Journal of New MusicResearch, vol.43, no.2, pp.183-201 (2014)