Keyboard Interface with Shape-Distortion Expression forInteractive Performance
Masa Ogata
National Institute of Advanced Industrial Science andTechnology (AIST)
Tsukuba, Ibaraki, Japanmasa.ogata@aist.go.jp

Masataka Goto
National Institute of Advanced Industrial Science andTechnology (AIST)
Tsukuba, Ibaraki, Japanm.goto@aist.go.jp

ABSTRACTIn this study, we set it as our goal to design an interactive
keyboard that changes its shape in response to the user’s
finger movements. The keyboard provides visual feedback in real time to make the user aware of the soundthey are producing, allowing the user to handle expressive control by swiping the finger after touching the keyboard. We implemented three variations of the shapedistortion design and its animation. We also implemented
three types of expressive control (volume change, pitch
bend, and timbre-mixing balance) for testing each shapedistortion design and for finding the best combinations
between shape distortion and expressive control.
We created the visual expression of a piano interface on a
two-dimensional planar surface (a touch display) and
expressive control with MIDI control. Unlike an ordinary
touch display that can only provide touch events, the system also detects finger hovering by tracking the threedimensional position of the finger with 0.1-mm accuracy
and at up to 100 Hz. With this accurate and frequent position, the keyboard changes its shape at up to 20 frames
per second in accordance with the proposed shapedistortion design. The number of fingers tracked is currently limited to one to prevent recognition error whentouching the keyboard. Thus, we have designed the current exploratory system for one-finger piano playing.
To test the concept of shape-distortion design, three keyboardists were asked to play music using the variations ofshape distortion with expressive control, and they discussed their preferred combination. On the basis of thisfeedback, we determined an efficient and preferred com-

This paper presents a design of a shape-distortion keyboard as a kind of visual feedback of finger input on a
piano interface for interactive performance. By tracking
the movements of a finger and by controlling MIDI output, the keyboard interface of the proposed design provides expressive control in accordance with the finger
movements during playing. Unlike an ordinary piano
interface with a static keyboard, the shape-distortion
design transforms the positions and shapes of keys, allowing swiping and shaking the finger to tune expressive
control. To test the designs of the shape distortion, we
implemented three variations of shape distortion andthree types of expressive control such as volume change,
pitch bend, and timbre-mixing balance, thereby enabling
us to find the effective combinations between them. In auser study, three participants tried these variations ofshape-distortion design and provided comments for feedback, improvement, and the future design of the interactive keyboard’s interface and performance.

1. INTRODUCTION
People use musical instruments as an interface to express
their musical imagination, and the resulting sounds candelight and inspire people. Conventional musical instruments are physical objects, so the function on each interface is physically limited. They cannot proactively
change their shape in accordance with user behavior. In
comparison with the conventional keyboard interface, avisual keyboard interface drawn on a display by software
has an advantage in designing multi-functional input andin giving interactive visual feedback in accordance withthe user’s input. Some research has focused on an assist
method and visual feedback of piano playing but not on
playing style and function. In addition, several interactive
systems have been proposed that augment the piano andprovide visual or haptic feedback to enhance the user
experience, but they leave the keyboard shape as is.
Our research question is on designing an interactive piano
interface for playing style and performance with soundeffects. An instrument whose shape could adapt to theuser in some way could open new possibilities for music
instruction or music expression. We aim to design aninteractive instrument that can change its shape. We areparticularly interested in designing a musical instrumentthat bends the visual feedback in accordance with theuser’s movements.

378

Figure 1. Responding to finger movements by distorting shape and changing color in accordance with
finger position. The marker of motion tracking wasworn on one finger.
2017 ICMC/EMW

Figure 2. Timing and animation sequence of shape-distortion design. This is a “wide distortion” pattern. The shape distortionand density of color are applied in accordance with the distance from the display. The target key to be touched next is in red.
The first target key is the second one from the left, and the second target key is the second one from the right.

bination between the shape distortion and expressive control. The responses also helped with other improvements,
and the future design of the interactive keyboard to enhance performance.

2. RELATED WORKOn the subject of novel piano playing systems, we canclassify previous approaches as novel interfaces or novel
visualizations. Novel interfaces are piano augmentations
that enable new gestures in performance, whereas novel
visualizations are piano augmentations that visually convey extra information to the player. Systems that are targeted at users of a certain skill level may be of either
type.
2.1 Interface ApproachThis section presents previous research related to the interface approach. The magnetic resonator piano [14] is amechanically modified piano keyboard that uses electromagnetic actuation to induce vibration, allowing novel
gestures like vibrato and after-touch volume control.
With the Kinect, PiaF [12] investigated the relationshipbetween pianist’s body movements and acoustic parameters while performing to find the way to augment performance. AirPiano [11] uses the finger tracker LeapMotion
to build a different musical touch interface for exploring
gesture motions and musical interactions. Several typesof sensors were combined to create a new hardware keyboard for musical applications [13]. It captures five separate dimensions of motion and key movements for richgesture input. The portable scanner hardware [15] attached on the piano keyboard turns the conventional piano into MIDI or Open Sound Control instruments. The
gesturally extended piano [17] is an augmented instrument controller to handle real-time audiovisual processing using a motion tracker. Real-time emotion detection [16] is done by motion tracking using a wireless sensor and an algorithm to classify the motions for mapping.
Virtual reality systems have also been developed to help
people play the piano. PianoTouch [1] and Mobile Music
Touch [2] are both wearable systems for piano instructionthat generate vibrations on the user’s fingers using vibraH E ARIN G THE SELF

tors attached to a glove. The glove instructs the user on
where to move his/her fingers while the user listens to themusic. Seaboard [3] is based on a different idea. The
keyboard changes from hard to soft material to let theuser play a continuous melody with one finger.
Unlike related projects, proposed system does not change
the user’s movements or augment the user’s body. We
selected the motion tracker to capture the player’s finger
motion correctly. Instead, this paper contributes novel
designs of shape-distortion keyboards that adapt to theuser.
2.2 Visualization Approach
Andantino [10] and Andate [9] are piano playing systems
that project animations and characters on top of the keyboard. They aim at pedagogical music visualization tokeep the user motivated. P.I.A.N.O. [5] provides a simple
music score visualization to let the users quickly understand where their fingers should move and to address thesteep learning curve of the piano. Musical notes in thescore are represented as blocks flowing down to the keyboard. These approaches augment the ordinary piano andleave the keyboard as is. In contrast, our approach changes the shape of the keyboard flexibly to enhance the styleof the user’s playing proactively.
Takegawa et al. [8, 6, 7] developed and evaluated a camera-based finger tracking system on a piano keyboard,which captures five color markers put on different fingernails to understand the user’s finger motions in real-time.
The metamorphosis hand [4] shows virtually extended
fingers that reach the target keys to make users feel likethey are playing the piano correctly regardless of the realpositions of their fingers. While our aim is closely related
to theirs, we chose to distort the keyboard shape rather
than the user’s body image.

3. CONCEPT
Shape-distortion design achieves the aim of this study:enhancing the style and performance of piano playing by
providing visual feedback as the shape distortion of a
piano interface that changes its shape. Unlike a conventional piano, the shape-distortion design makes the key-379

board interactive and flexible in accordance with the
movements and touches of a user’s finger. We created
new designs of interactive piano interfaces powered bysensing techniques and the interactive design of visualfeedback. In shape-distortion design, the goal was for the
shape to change interactively to generate visual feedback
that enhances the user’s ability to play the piano.
We simplified the design of shape distortion and implemented visual feedback written in software, and we conducted a test of the design concept with keyboardists. The
playing method was limited to assisted playing, whereinusers touched the next key to play along with a musicscore. The music score was set in advance and played on
the software when the users’ finger touched the keyboard.
Our goals for the shape-distortion design for an interactive piano interface on a touch display were as follows:(1) to create a design of a keyboard interface using shape
distortion as visual feedback of control for intuitive andinteractive playing, (2) to connect expressive control totypes of shape distortion to find effective and preferredcombinations through the user study, and (3) to utilize the
keyboard interface design of interactive playing, allowing
users degrees of freedom in their hand and body.
3.1 Types of Shape-Distortion
We prepared three types of shape distortion by imaging
animation as visual feedback to express the results oftuning expressive control. The frame rate of graphic images is about 15–20 Hz. The animation and transitionstyles of each variation are shown in Figure 4 with thetime sequence going left to right. The key where the next
note should be played on the touch screen is called the“target key” in the following paragraphs.

Figure 3. Setup of motion tracking camera and display
arrangement. It was also used for the actual user study.

3.1.1 Wide Distortion
The target key moves under the finger when touched andwhile keeping its width the same. Other keys to the rightand left of the correct key are widened or narrowed by the
distortion. Each white and black key is drawn with equalintervals. Thus, both the right and left parts of the keyboard shrink in accordance with the finger position.
3.1.2 Curve Distortion
Only the target key is distorted and moved over the keyboard. The target key overlays other keys and moves inaccordance with the finger position. Therefore, just one
key is moved under the finger, while the others are fixed.

Figure 4. Three types of effect (Wide, Curve, and Fat) are shown with four levels of distortion ratio. Each effect and finger position is captured by a camera above the display. The distortion ratio is estimated, and the finger height is roughly indicated foreach situation.

380

2017 ICMC/EMW

3.1.3 Fat Distortion
The target key is widened to cover the position of thefinger while keeping the target key in its original positionin the key order. Half of part of the keyboard shrinks inaccordance with the finger position. Thus, this is thesame as a wide distortion for half the area of the keyboard.
3.2 Timing and Animation
Matching the timing and animation generation are required to achieve an interactive keyboard interface on atouch display. We designed a shape- and color-changing
sequence in accordance with the ratio of the depth position of the finger (Figure 2). All visual feedback andshape distortions are drawn as animations to present thevisual feedback and reaction in response to the user behavior. We designed three types of shape distortion to fit
the “target” key—the key that is highlighted in anticipation of the user’s next press—for the user’s finger position. Our implementation of the shape-distortion algorithm makes it important that the vertical finger position
from the display (height from the display surface) generate visual feedback with animation.
3.3 Finger, Sound, and Score
The shape-distortion design presents an interface thatchanges its shape in accordance with the finger positionin three dimensions. Parallel axes of the display (x and yaxes) make points on the keyboard for the finger to touch.
From the threshold of height (z axis) to zero distance, thekeyboard shape distorts in various ways when the finger
comes close to the keyboard. A sound is made when thefinger touches a key on the keyboard.
The system has a piano score prepared in advance. The
system presents the recorded score on the keyboard andmoves the next key to be touched under the user’s finger
when he/she is about to touch the keyboard. Like an electric lighted keyboard (which turns the light on the key totouch), an indication is shown on the keyboard to direct
the player to touch the correct key to continue playing thescore. The indication turns both white and black keys into
red.
3.4 Limitation
First, we tested the proposed design by implementing asystem that can let a player use one finger to interact withthe piano interface. We set the limitation as one finger
input to test the interactive system and the distortion design. This limitation of one finger input is natural anddoes not cause any problem when performing a monophonic solo instrument on a synthesizer. This also enables avoiding sensing errors by the finger tracker when
too many fingers are on the marker(s). In this system, atouched state is recognized as a touch sound on the display, but an untouched state is recognized as the distancefrom the display. Thus, the user has to raise the finger
over the threshold to make sure that the system detects
the finger has left the key.

H E ARIN G THE SELF

4. IMPLEMENTATIONThe system consists of a touch display, finger tracker,
and software interface. Figure 3 shows the setup of theenvironment for this system.
4.1 Setup
For sensing the user’s finger, we placed five OptiTrack
FLEX 13 cameras (an IR camera tracking system) above
the touchscreen to measure finger gestures and heightfrom the screen. Each camera is placed so as not to interfere with the other cameras because reflection on thetouch display may generate noise and lead to misrecognition of the finger position. We designed an IR marker tobe worn on the finger for finger tracking. The touch
screen display (EIZO T2381W) is used for both displaying the keyboard and measuring the timing of each touch.
To increase robustness, we also used acoustic sensing
with a contact microphone installed in the back of thedisplay to sense touch sound on the display, thereby ensuring every touch is captured.
For output, the piano interface is projected on a display
placed on a table. We selected the MIDI interface to generate sound in accordance with keyboard touches. Because the MIDI interface implemented in Windows suffers from noticeable delays, we used a MIDI interface(Roland Edirol SD-90 Studio Canvas) that outputs soundwithin a time short enough for humans to not notice any
delay.
4.2 Tracking
The tracking system should be fast and reactive to theuser’s finger motion because playing piano requires active and fast movements. Measuring the finger position
and touch is one of the requirements of this system. Our
first prototype used a Microsoft Kinect version 2 device
to capture depth images and to extract the finger position,
but it did not work due to the slow response time andlow-resolution image. The OptiTrack cameras and accompanying software provide the exact position (up to
0.1 mm) and a high frame rate (over 100 Hz). The software gives the position of the IR marker worn on the finger.
The finger position is used to determine the finger heightfrom the screen. To avoid an error where multiple keypresses are detected during a single keypress event, thefinger must be moved above the threshold (about 5 cm
from the screen surface) before going to the next touch
position. Without doing this, the keyboard is locked, andit sustains the current note.
Only during timbre-mixing balance, we selected thesound as “StringEnsemble1” (Left side) and “Piano”(Right side) for mixing. Other types of expressive control
used “Lead2Sawtooth” for the sound.

381

Figure 5. Implementation and use of pitch bend. The time goes from left to right while the finger moves around the correct
key position. The sound from MIDI varies from four semitones below to four semitones above. When too sensitive, users canchange the rate of magnification. The pitch changes in accordance with the vertical position of the finger and reflects the distance from the indication key.

4.3 Expressive Controls
We also implemented three types of expressive control
using MIDI control. Thus, the keyboard interface works
like the MIDI synthesizer, but the control method differsfrom the ordinary MIDI synthesizer because of the shape
distortion using an integrated control method with thefinger movements. Users can select the reference point ofexpressive control as the center of the keyboard screen or
the position of the target key. If users select the center of
the keyboard as the reference point, the minimum andmaximum of the control level is set to the left and rightedge of the keyboard screen.

5. USER STUDY
We asked three participants to play with our keyboard
interface to evaluate the playing style using shape distortion with expressive control. As described in the Concept
chapter, the music score (Edward Elgar's Pomp and Circumstance, Op.39) was already set in advance, so theparticipants played the music by tracing in accordancewith the music score. We also prepared the printed score
for reference, but they did not need to see it because therequired keys were highlighted on the screen as aforementioned.

following, we summarize the comments from all participants for each distortion method:
5.1.1 Wide Distortion
Because the side distortion changes the shape of thewhole keyboard, it was useful for expressive control thatuses absolute values such as volume control. Timbremixing balance was also evaluated when used with thereference point on the keyboard. The participants were
not satisfied with the pitch bend because the sense of
shape distortion does not match the expressive control.
5.1.2 Curve Distortion
The curve distortion is only the shape-distortion design
which matches the expressive control of the pitch bendwith the reference point of the target key. Because of thetype of shape distortion, the curve changes only the shapeof the target key, and the other keys are fixed. Thus, theparticipants preferred using curve distortion with the ref-5.1 Results of User Study
In figure 6, we show the result of an evaluation for eachcombination of shape distortion and expressive control.
We used a symbol to express the result rather than a numerical value (such as a Likert scale) to describe the efficacy and intuitiveness of playing style with them. The
best match was “◎,” which indicates satisfaction, and thelowest was “☓,” which indicates dissatisfaction. In the382

Figure 6. Result of user study with evaluations for eachcombination of shape distortion and expressive control.
The symbols mean the level of efficacy and intuitiveness of the combination. The screen center or target key
under the symbol means the effective selection of thereference point of expressive control.
2017 ICMC/EMW

erence point of the target key and pitch bend, which controls the relative pitch from the pitch of each touched key.
5.1.3 Fat Distortion
The participants reported that fat distortion is similar towide distortion because half of the keyboard shrinks. Because of dynamic change in the keyboard design, it does
not match the pitch bend. Furthermore, expressive controlof volume change and timbre-mixing balance were notevaluated to compare with wide distortion.
5.2 Discussion
We found that the efficient and preferred combination
depended on the reference point and the type of expressive control. Because of its unusual shape, Curve distortion was expected to be the worst candidate for the shapedistortion design before the user study, but it was actually
evaluated as being the best for combinations with pitch
bend control because of the ease of playing and becauseof the sense of playing style in using the finger position
for the control function. Keyboardists usually use theright hand for such a monophonic performance and theleft hand to move the pitch bend wheel/lever, but in ourdesign, they can take the advantage of just using one finger to control both the keyboard and the pitch bend.

6. CONCLUSION
We designed and implemented a conceptual piano interface that uses shape distortion. With accurate finger
tracking and an interactive interface design, the positions
and shapes of keys flexibly change in response to finger
movements. We tested three types of shape-distortion
design with three types of expressive control to find an
efficient and preferred combination of two of them forplaying style. We found that the planar keyboard interface with graphical feedback of distortion design had an
advantage in intuitively performing expressive control
such as pitch bend just by using one finger. Typical expressive synthesizer performances with pitch bend usually require both hands, one for pressing a key and the other for controlling the pitch bend wheel. Participants in auser study classified and evaluated their preferred combination, and our future design of the keyboard interface
will be changed on the basis of their comments.
Acknowledgments
This work was supported in part by JST ACCEL Grant
Number JPMJAC1602, Japan.

7. REFERENCES[1] Huang, Kevin, Ellen Yi-Luen Do, & Thad Starner.
PianoTouch: A wearable haptic piano instruction systemfor passive learning of piano skills. Proc. ISWC 2008,
pp. 41–44.
[2] Huang, K., Starner, T., Do, E., Weiberg, G., Kohlsdorf,D., Ahlrichs, C. & Leibrandt, R. Mobile music touch:
H E ARIN G THE SELF

mobile tactile stimulation for passive learning. In Proc.
CHI 2010, pp. 791–800.
[3] Lamb, Roland, & Andrew Robertson. Seaboard: a New
Piano Keyboard-related Interface Combining Discrete
and Continuous Control. In Proc. NIME 2011, pp. 503–
506.
[4] Ogawa, N., Ban, Y., Sakurai, S., Narumi, T., Tanikawa,T. & Hirose, M. Metamorphosis Hand: DynamicallyTransforming Hands. In Proc. AH 2016, No. 51.
[5] Rogers, K., Röhlig, A., Weing, M., Gugenheimer, J.,
Könings, B., Klepsch, M., Schaub, F., Rukzio, E.,
Seufert, T. & Weber, M. PIANO: Faster Piano Learningwith Interactive Projection. In Proc. ITS 2014, pp. 149–
158.
[6] Takegawa, Yoshinari, Tsutomu Terada, and Shojiro
Nishio. Design and Implementation of a Real-timeFingering Detection System for Piano Performances.
Proc. ICMC 2006, pp. 67–74.
[7] Takegawa, Y., Tsukamoto, M., Terada, T. & Nishio, S.
Mobile Clavier: new music keyboard for flexible key
transpose. In Proc. NIME 2007, pp. 82–87.
[8] Takegawa, Y., Tsutomu Terada, and Masahiko
Tsukamoto. A piano learning support system
considering rhythm. Proc. ICMC '12, pp. 325–332.
[9] Xiao Xiao, Basheer Tome & Hiroshi Ishii. Andante:
Walking Figures on the Piano Keyboard to Visualize
Musical Motion. Proc. NIME '14, pp. 629–632.
[10] Xiao, Xiao, & Hiroshi Ishii. Inspect, Embody, Invent: A
Design Framework for Music Learning and Beyond. In
Proc. IDC 2016, pp. 5397–5408.
[11] N. d’Alessandro, J. Tilmanne, A. Moreau, & A. Puleo.
AirPiano: a multi-touch keyboard with hovering control.
Proc. NIME 2015, pp. 255–258.
[12] A. V. Zandt-Escobar, B. Caramiaux, & A. Tanaka. PiaF:
a tool for augmented piano performance using gesturevariation following. Proc. NIME 2014, pp. 167–170.
[13] McPherson, A. P., & Kim, Y. E. Multidimensional
gesture sensing at the piano keyboard. Proc. CHI 2011.
pp. 2789–2798).
[14] A. McPherson. Techniques and circuits forelectromagnetic instrument actuation. Proc. NIME 2012.
[15] A. McPherson. Portable Measurement and Mapping ofContinuous Piano Gesture. Proc. NIME. 2013. p. 152–
157.
[16] M. Ben-Asher and C. Leider. Toward an emotionallyintelligent piano: real-time emotion detection andperformer feedback via kinesthetic sensing in piano
performance. Proc. NIME 2013, pp. 21–24.
[17] W. Brent. The gesturally extended piano. In Proc.
NIME, 2012.

383