Touch-typable VR piano that corrects positional deviation offingering based on music theory
Chika Matsusue
Nagoya Institute of Technology
Gokiso-cho, Showa-ku, Nagoya, Japanchika@center.nitech.ac.jp

Kenji Funahashi
Nagoya Institute of Technology
Gokiso-cho, Showa-ku, Nagoya, Japankenji@nitech.ac.jp

Shinji Mizuno
Aichi Institute of Technology
1247 Yachigusa, Yakusa-cho, Toyota, Japans_mizuno@aitech.ac.jp

Figure 1: Appearance of the system

ABSTRACTIn this paper, we propose a novel virtual piano system to assist
composition, that corrects deviation of a position based on musictheory when a user strikes a wrong key. It means that this systemestimates a correct melody from deviated fingering due to the nonphysical keyboard. When experienced performer comes up with agood melody, his hands usually move by themselves to play it. Our
proposed VR piano does not have any physical keyboard; it estimatesa correct key when user’s fingering is deviated. Furthermore, if theweight of music theory is adjusted higher than the weight of the user’s
fingering position, this system changes to easy-playable VR piano.
Even by hitting a key randomly, it will produce an appropriate melody
based on music theory by taking a bit of fingering into account.

CCS CONCEPT• Computer Applications → Art and Humanities

1INTRODUCTION

When an experienced performer comes up with a good melody, hishands play it by themselves on the keyboard. But it is obviously
difficult to bring an instrument anytime, anywhere because an
actual instrument is large. There are already some researches about
virtual piano like, for example, its design via single-view video [1].
There are also many piano applications for smartphone and tablet.
Although you can see a keyboard, it is difficult to strike a key
without any haptic feedback. Therefore, experienced performer’s
hands can not move to play the melody by themselves. On theother hand, many composition assistance systems are proposed, forexample, songwriting support system [2], that suggests a melody
by a curve drawn on a card. However, this is not a traditional style

to perform a melody. Our VR proposed piano does not have any
physical keyboard either. When the user moves his hand on thetable in such a way to play a melody with it, and the fingering isdeviated from the key that the user wants to strike, it revises thedeviation to a correct key based on music theory[3]. This system
obtains the user’s fingering with a depth camera or a motion sensor,
then it estimates the melody that the user wants to play.
Furthermore, if the music theory weight is higher than the user’s
fingering position weight, this system changes to easy-playable VR
piano. Even by hitting a key randomly, it will produce anappropriate melody based on music theory by taking a bit offingering into account.

2TOUCH-TYPABLE VR PIANO

To play the VR piano on a table, it is necessary to obtain thefingers motion. Although it is easy to get a hit on a tablet, theperformance range is limited by the screen size. The player shoulddrag the keyboard on the screen for example. Our proposed system
obtains fingertip positions and joints of a hand through Leap
Motion. No keyboard image is shown on the table, but anexperienced pianist can move his fingers without seeing one.
However, it is still difficult to recognize their actual finger position
to strike a key for next note like there is no haptic feedback fromphysical key. Of course, just by playing on a table and sounding
notes in their mind, there is no problem. It means that thereliability of fingering data is not so high. Fingering is sometimes
deviated from the wanted key to be played. Therefore, music theory
supplements it on our system. The method to correct a fingering
deviation using music theory is as followed:

SIGGRAPH 2017 Posters, August 2017, Los Angels, CA, USAC.Matsusue et al.

h(a) A part of played melody (for ratio 60:40)

×(b) A part of corrected melody (with ratio 60:40)

Figure 2: Probability for keystroke position(c) A part of notes struck randomly (for ratio 45:55)
×(d) A part of estimated melody (with ratio 45:55)
Figure 4: Experimental results
×

Figure 3: Probability based on music theory
Step1: Probability for keystroke position
The key that the user wants to hit must be close to the stroke position,and the relation between it and the previous stroke is reliable. We
consider an isosceles triangle with a base length of 2|xi – xi-1| and area of 1(Fig. 2) for the keystroke position (xi, yi) obtained at time ti. We define Pki(k is a letter note as C, D, E, F, G, A, B, considered for three octaves) as
the area shown in Fig. 2-(1). Then for example, Pdi is a probability tojudge that the D key was struck at time ti.
Step2: Probability based on music theory
We consider a chord tone as music theory, a probability of a tone
according to the chord should be higher at the stroke time or in the bar.
When the code is CM7 for example, a probability Qki for each key isdefined equally as shown in Fig. 3-(1). We consider just the keys that arecovered in the triangle mentioned above, and sum of these areas at time ti
is 1. When the code is BbM7 for another example, a probability Qki isdefined as 0.25 or 0 as shown in Fig. 3-(2).
Step3: Determination of note
Finally, the probability for each note is defined as Rki = wRPki + wQQki,
where wR and wQ are weights that satisfy wR + wQ = 1. The maximum Rkifor every k at time ti is selected, then the letter note k is sounded. Inaddition, if |xi-1 - xi| is greater than a threshold, the center of the key
position of note k is shifted to the striking position xi every processing
time, in other words, x axis origin is shifted to adjust xi as the center of k.
When the weight of music theory wQ is 0 and the weight of user's
fingering position wR is 1, just the note at the stroke position is selected
and sounded. When wQ is adjusted larger than wR, this system can beused as an easy-playable VR piano.

3EXPERIMENT

We used Leap Motion and acrylic board to develop our new VR piano asshown in Fig. 1. Although we thought that a user plays this without
seeing, keyboard CG and virtual hand were shown on a screen as shownin Fig. 1 (right) to confirm deviation correction and keyboard shiftaccording to the correction. This screen image was also used to decideinitial hand position. In a first experiment, we assumed that a part of a2

melody of "Do Re Mi" was played, the chord progression was Dm7, Dm7,CM7, and then FM7 for 4 measures from diatonic chords of C major key.
Diatonic chords are the chords that are derived from the notes of a key.
Then this system corrected the fingering with the reliability ratio as
fingering:theory = 60:40. If assuming that the center of a key is struck,when |xi-1 - xi| is 2 degrees, that means neighbor key, and when wP is
greater than or equal to 45%, fingering is adopted. When |xi-1 - xi| is 3
degrees, and when wP is greater than 73%, fingering is also adopted
certainly, however to adopt theory certainly, wR should be less than 64%.
Actually a hand was moved on this system slightly deviated form themelody (Fig. 4-(a)). In a second experiment, we assumed that a chord
progression was the progression as CM7, CM7, G7, and then CM7 for 4
measures. The hand was moved almost randomly (Fig. 4-(c)), then it
estimated the appropriate melody with the ratio 45:55. Fig. 4-(b) and (d)show the results. The melody that fingering was correcting was almost
like the "Do Re Mi". And the melody that was estimated from random
fingering felt us that it was not a random note sequence, but real music.

4CONCLUSION

In this paper, we propose a novel virtual piano system that user can playwithout seeing keyboard. Now we are implementing next element ofmusic theory as scale that is a specific set of notes, and transition of notes
is sometimes according to the order in a scale. In the future, we should
estimate a principal key and chord progression, it means the chord at
stroke time or in that bar. This current system selects only note that is thekey or the neighbor key. We should implement other elements such astension note that is non-harmonic note added onto basic chord, etc. The
initial position should be also estimated automatically from the very firststrokes.

REFERENCES[1] Chia-Hung Yeh, Wen-Yu Tseng, Jia-Chi Bai, Ruey-Nan Yey, Sun-Chen Wang andPo-Yi Sung 2010. Virtual Piano Design via Single-view Video based on Multifinger
Actions Recognition. 3rd International Conference on Human-Centric Computing,pp1-5(Aug. 2010), DOI:http://dx.doi.org/10.1109/HUMANCOM.2010.5563348[2] T.Iwata, H.Sawano, T.Ikeda and T.Hishida 2012. A Songwritibg Support Systemwith a Card. ACM SIGGRAPH Asia 2012 Sketch&Posters, Singapore, Article 24(Dec. 2012), DOI: http://dx.doi.org/10.1145/2407156.2407183[3] http://www.tonalcentre.org/