To appear in: Proceedings of the 9th International Conference on Music Perception and Cognition (ICMPC9), Bologna, August 2006

Unobtrusive practice tools for pianists
Werner Goebl (1)Gerhard Widmer (1&2)(1) Austrian Research Institute for
Artificial Intelligence
Vienna, Austria
http://www.ofai.at/music(2) Department of Computational Perception
University of Linz
Linz, Austria
http://www.cp.jku.at
tational implementation of a MIDI-based performance
visualization system has been presented by Smoliar, Waterworth, & Kellock (1995). The displayed performanceparameter include onset and offset timing, dynamics, articulation (referring to tone length), and tone onset asynchronies. However, the paper lacks a technical descriptionof how these parameters were computed and implemented,
but it becomes evident that this system works in an offlinemanner (computing the displayed parameters after recording).
The elaborated MIDI-based visualization system“comp-i” (Hiraga, Mizaki, & Fujishiro, 2002) depicts performance information (onset and offset timing, dynamics,pitch) in a 3-dimensional space. Another approach aimed to
explicitly exemplify higher-level performance information
was provided by Hiraga & Matsuda (2004). They calculateand display local tempo change, local articulation, and local dynamics change. Both of the just mentioned systems
work offline (Hiraga, Mizaki, & Fujishiro, 2002, Hiraga &Matsuda, 2004). More recent developments aim at developing computer software that allows relatively fast access toperformance information, such as tone intensity or timing(MIDIator software from the Piano Pedagogy Research
Lab in Ottawa1).
High-level approaches aim to impose specific models
of emotions and emotional playing on music students via
feedback about the goodness of fit of the human performance to the model (Juslin, Friberg, Schoonderwaldt, &Karlsson, 2004). However, such an approach seems problematic because it makes an aesthetic judgment by predefining a target performance on the basis of an artificial
cognitive model.

ABSTRACTThis paper proposes novel computer-based interfaces forpiano practicing. They are designed to display in real timecertain well-defined sub-aspects of piano playing. They areintelligent and unobtrusive in that they adjust automatically to the needs of the practitioner so that no other interaction is needed than moving the piano keys. They include1) a pattern display, finding recurring pitch patterns anddisplaying expressive timing and dynamics thereof, 2) achord display, showing timing asynchronies and tone intensity variations of chords tones, and 3) an acoustic pianoroll display that visually models the acoustic piano tone
from MIDI data.

Keywords
Piano Practice, Performance Visualization, Music Education

INTRODUCTION
There is an increasing interest in the possible roles anduses of computers in music education (Webster, 2002), andthe technical facilities needed are easily available to virtually everyone. However, there are still only few approachesfor developing computer-based tools that are successfully
used in every-day music performance practice. As an example, we mention refined spectrogram representations
with fundamental frequency trackers that are used in singing lessons (see e.g., the VOXed system Welch, Himonides, Howard, & Brereton, 2004).
In piano performance, visual feedback from conventional piano roll displays has proved to be a helpful meansin piano instruction (Riley-Butler, 2001, 2002). A compu-AIMS
In this paper, we propose piano practice interfaces that
support the pianist in enhancing the effectiveness of his or
her daily practice load. They are not meant to be THE solution to all practice problems; instead, they focus deliberately on certain sub-tasks of piano playing. They provide inreal time immediate visual feedback via the computer
screen to the performer and require a MIDI-compatible
piano. They are intelligent in that they “listen” to the pianist’s playing and decide themselves what to display.

Proceedings of the 9th International Conference on Music Perception &Cognition (ICMPC9). ©2006 The Society for Music Perception & Cognition (SMPC) and European Society for the Cognitive Sciences of Music(ESCOM). Copyright of the content of an individual paper is held by theprimary (first-named) author of that paper. All rights reserved. No paper
from this book may be reproduced or transmitted in any form or by anymeans, electronic or mechanical, including photocopying, recording, or by
any information retrieval systems, without written permission from thepaper’s primary author. No other part of this book may be reproduced ortransmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information retrieval system, withwritten permission from SMPC and ESCOM.

11

http://www.piano.uottawa.ca, seen in May 2006

Figure 1 Pattern Display. The left panel shows a 6-tone pattern in a pitch–time space just after its last tone has been played. Circle sizeand color indicate dynamics (MIDI velocity), horizontal placement the timing deviation relative to the timing of the corresponding pattern
tone of the previous pattern cycle (displayed as shaded disks). On the top-right, the auto-correlation pattern is shown (correlation coefficients against shift amount). The bottom-right panel depicts a screen-shot of the frugal beattracker (against time in s), with the present
moment marked by “now.” Circles denote played tones, (red) lines tracked beats, and shaded lines expected beat onsets. On the right, thefuture expected beat is outlined.

forming fast accompaniment passages, such as Alberti bass
figures frequently found in the Classical piano literature.
For example, take a bass tone followed by two mediumpitched tones, all three alternated by a higher-pitched tone
– therefore a six-tone pattern. It repeats several times withslightly varying pitches, but with the same pitch contour.
The student wants to know what it is that makes these figures sound irregular, dull, or the opposite – swinging andvibrant.
Our interface automatically identifies the above pattern
after typically 2 or 3 repetitions and displays the timing anddynamics of its six tones in real time while the student ispracticing. The student can therefore react on his or her
irregularities and shape the performance of that patternaccording to her taste. We describe in the following thethree processing levels of this pattern interface.

They are unobtrusive in that they do not require any
direct interaction of the user (the pianist) via computer keyboard or mouse. We argue that any change of haptic
modality (from piano keyboard to computer touch pad)would disrupt the flow of practice considerably. The proposed practice interfaces could run permanently in a practice session, getting sometimes the pianist’s attention (e.g.,when she is deliberately scrutinizing her own chord play),and sometimes none at all.
The proposed practice interfaces currently consist of
three parts which we will describe in the following:
1. A pattern display that identifies recurring pitch sequences and shows timing and dynamics deviations of
the found pattern;
2. A chord display that displays timing asynchrony andtone intensity balance of chords (sequences of tones
within a certain time window); and3. An acoustic piano roll representation that visuallymodels properties of the piano sound including decay
and pedal interaction.
The current version is implemented in JAVA 1.5 programming language and therefore runs on multiple computer platforms.

Pattern Finding Algorithm
The pattern finding algorithm uses autocorrelation todetect a pattern. Sequences of relative pitch (pitch differences) are correlated to each other repeatedly, each time
shifted one element further apart.2 The smallest shift at
which the correlation coefficient has a peak and is beyonda certain threshold is taken to be the cycle length of thePATTERN DISPLAY

2Let’s assume the following practice situation: a piano
student wants to improve his or her technical ability in per-2In case of a chord (more than one tone within 70 ms), the mostprominent tone (in terms of MIDI velocity) is considered forcalculation.

tions are displayed against the beat estimate from the beattracker.
A tone occurring too early is displaced leftwards withthe corresponding previous tone of the pattern shown greyin the background. A more common way to display timing
would be to show inter-onset interval (IOI) timing. However, we strongly prefer the present display over plotting
IOIs because every IOI value stems from two tone onsets,
so it would be hard for the performer to disentangle thesetwo pieces of information. A screenshot of the componentsof the pattern display is shown in Figure 1. The left panel
shows a 6-tone pattern just after its last tone has been
played with pitch on the y axis (the middle-C key ismarked grey) and time on the x axis. Circle size and color
correspond to dynamics (MIDI velocity), horizontal placement to timing deviation relative to the timing of the corresponding pattern tone of the previous pattern cycle (displayed as shaded disks).
On the top-right, the auto-correlation pattern is shown(correlation coefficients against shift amount), indicating apattern period of six tones. The bottom-right panel depicts
a screen-shot of the frugal beattracker (against time in s),
with the present moment marked by “now.” Circles denote
played tones, red (in case of black and white print: dark)lines successfully tracked beats, grey lines expected beatonsets. On the right, the future expected beat is outlined.

Figure 2 The Chord Display (voice number against time in ms)
shows onset asynchronies of chords and the dynamics of eachtone. Mean first-note lead and maximum chord spread are printed
on top of the panel.

Chord Display
The second aspect of piano performance visualized by
the proposed system concerns timing asynchronies andtone intensity balance of chords. Pianists almost never play
nominally synchronous score notes entirely simultaneously.
Studies on expressive piano performance report systematic
trends in these asynchronies (Palmer, 1996). In particular,when melody tones are emphasized (played louder), theironsets typically occur around 30 ms ahead of the otherchord tones (Goebl, 2001). Apart from that, the timbral
color of a chord may be shaped deliberately by the performer by altering the intensity balance of the chord tones(Neuhaus, 1973).
The proposed chord display detects potential chords in
the MIDI data stream, calculates the first-tone lead and thechord spread, and displays the tones of the chords according to their pitch order (vertical axis) and time (horizontalaxis). The intensity of the tones are reflected in size andcolor of the displayed disks (the louder, the larger and themore red on the same scale as in the pattern display). The
leading tone is indicated by a black margin. A screenshot
of a chord display example is shown in Figure 2.
A chord is defined by a simple heuristic: tones belong
to a chord when each individual chord tone is no more than
70 ms apart and the total spread does not exceed 300 ms.
These figures were approximated from experience with alarge corpus of piano performance data.

pattern. In order to provide a more stable behavior of thepattern finding algorithm, cycle length is kept in a buffer ofwhich the most frequent value (mode) is considered. The
phase of the pattern is determined by the relative tone intensity within the pattern, the loudest tone being the first.
Through this definition, the performer can re-adjust thepattern start as desired simply by emphasizing the first noteof a pattern.

Frugal Beat Tracking
Parallel to the pattern finding algorithm, a simple (frugal) beattracker determines a fundamental tempo, fromwhich estimates for future beats can be inferred. For programming efficiency, a very simple approach was followed(Pardo, 2004), although we assume that more elaborate
beat tracking algorithms (such as Dixon, 2001) are moreapt for this purpose. They will be integrated into the present system in the near future.

Display
As soon as a pattern has been established, its constituent tones are shown in a pitch–time space. Individual tones
are displayed as colored disks varying in size and intensityof color (reddishness) with tone intensity. The higher theMIDI velocity value, the more the yellow of soft tones
turns into red and the larger the disk gets. Timing devia-3

Figure 3 Acoustic Piano Roll. An excerpt of a performance of Schubert’s G-flat Major Impromptu is shown (pitch against time in ms).
Background shading of the panel corresponds to pedal press; color saturation of the tone bars depicts acoustic decay of the piano tones;
thinner decay bars indicate tones that are still sounding after note off due to pedal press.

Whenever a tone is played (i.e. when MIDI data is received), the display is updated and displays the last chordplayed. It works therefore in real time. As simple as thistool is, as explicit is its statement to the performer: It immediately exemplifies the relation of vertical dynamics
differentiation to voice anticipation or lag.

intensity (on a color map between yellow and red), the decay of the piano tones (represented by color saturation),and the interaction of the right pedal and the tones thatsound during pedal press (represented by prolonging thedecaying tones with slightly thinner bars). The right pedal
itself is shown as grey shading of the background.
To model the tone decay of a piano, a set of piano
samples recorded from a computer-controlled grand piano(Bösendorfer SE290) was analyzed in terms of their loudness decay over time. 25 different pitches distributed overthe whole keyboard were played back on the computercontrolled piano in 5 different dynamic levels (MIDI velocity units from 30 to 110 in steps of 20), each tone sounding
for a minimum of 5 seconds (for the low tones 10~s).
Loudness representations of these 125 samples were measured in sone according to the Zwicker model (Zwicker &Fastl, 1999) (implemented by Pampalk, Rauber, & Merkl,2002) and used to interpolate decay functions for allpitches and dynamic levels (MIDI velocities). Essentially,
lower tones decay later than higher pitches; softer tones
decay faster than loud ones. These data were linearlymatched to saturation of an HSV color space; thus, a decaying tone is losing color with time and turning more andmore into white.
As the other parts of the proposed system, the acoustic
piano roll works in real time. To avoid showing always thefull range of possible tones of the piano (usually 88 keys
from A0 to C8), the interface takes care of dynamically

ACOUSTIC PIANO ROLL DISPLAY
Piano roll displays are very common and can be foundin every sequencer software package nowadays. They usually represent onset and duration of played tones throughthe position and extent of bars in a pitch–time space. By
displaying this, they provide important note information;
however, essential data of piano performance is left out,that is e.g., pedal information (especially from the rightpedal), dynamics of each tone, and the interaction between
pedals and the sounding tones (These data are usuallyshown in separate channels or on demand; e.g., by clicking
on a note bar, its velocity is shown.) Here, we decided toinclude all performance data derived from a piano into asingle comprehensive piano roll representation. A screenshot of this interface is shown in Figure 3, displaying anexcerpt of a performance of Schubert’s G-flat Major Impromptu.
In the pitch–time space, we integrated a visual approximation of what is sounding at a given time on the basis of MIDI-like data from a piano performance. This includes – beyond displaying the onset and offset of tones
with bars – the dynamics of each tone, represented by color4adjusting the displayed pitch range on the vertical axis tothe present performance.

Hiraga, R., Mizaki, R., & Fujishiro, I. (2002). Performance visualization: a new challenge to music through visualization, Proceedings of the tenth ACM International Conference on Multimedia, Juan-les-Pins, France (pp. 239–242). New York: ACM
Press.
Juslin, P. N., Friberg, A., Schoonderwaldt, E., & Karlsson, J.
(2004). Feedback learning of musical expressivity. In A. Williamon (Ed.), Musical Excellence. Strategies and Techniques toEnhance Performance (pp. 247–270). Oxford: Oxford University
Press.
Neuhaus, H. (1973). The Art of Piano Playing. London: Barrie &Jenkins.
Palmer, C. (1996). On the assignment of structure in music performance. Music Perception, 14(1), 23–56.
Pampalk, E., Rauber, A., & Merkl, D. (2002). Using smoothed
data histograms for cluster visualization in self-organizing maps.
In J. R. Dorronsoro (Ed.), Proceedings of the International Conference on Artificial Neural Networks (ICANN'02), Madrid, Spain(pp. 871–876). Berlin: Springer.
Pardo, B. (2004). Tempo tracking with a single oscillator, Proceedings of the 5th International Conference on Music Information Retrieval (ISMIR2004) (pp. 154–157). Barcelona: Universitat
Pompeu Fabra.
Riley-Butler, K. (2001). Comparative performance analysis
through feedback technology, Meeting of the Society for Music
Perception and Cognition (SMPC2001), August 9–11, 2001 (pp.
27–28). Kingston, Ontario, Canada: Queen's University.
Riley-Butler, K. (2002). Teaching expressivity: An aural–visualfeedback–replication model, ESCOM 10th Anniversary Conference on Musical Creativity, April 5–8, 2002. Liège, Belgium:
Université de Liège.
Smoliar, S. W., Waterworth, J. A., & Kellock, P. R. (1995). pianoFORTE: A system for piano education beyond notation literacy, Proceedings of the ACM International Conference on Multimedia, San Francisco (pp. 457–465). New York: ACM Press.
Webster, P. R. (2002). Computer-based technology and musicteaching and learning. In R. J. Colwell & C. Richardson (Eds.),
The New Handbook of Research on Music Teaching and Learning(pp. 416–439). Oxford: Oxford University Press.
Welch, G. F., Himonides, E., Howard, D. M., & Brereton, J.
(2004). VOXed: Technology as a meaningful teaching aid in thesinging studio. In R. Parncutt & A. Kessler & F. Zimmer (Eds.),
Conference on Interdisciplinary Musicology, April, 15–18. Graz,Austria: University of Graz.
Zwicker, E., & Fastl, H. (1999). Psychoacoustics. Facts andModels (Second updated ed.). Berlin, Heidelberg: Springer.

FINAL REMARKSThis paper has presented visualization tools to be used
in a pianist’s every-day life at practicing piano. They have
not yet been extensively tested by pianists in practice. In
future work, usability studies will show advantages anddrawbacks of the present tools which we will use for further improvements. Foreseen extensions of these interfaces
will include other performance parameters (such as tonelength), improvement of pattern finding and beat-tracking
algorithms, and the identification of other pianistic subtasks. Furthermore, immediate display of touch characteristics could be realized with new computer-monitored instruments as, e.g., found in the latest version of Bösendorfer’s computer-controlled piano (“CEUS”) that provides
continuous key position data.

Acknowledgments
This research was supported by the Vienna Scienceand Technology Fund (WWTF, project CI010 “Interfacesto Music”) and by the Austrian Science Fund (FWF,START project Y99-INF and a Schrödinger Fellowship tothe first author, J2526). The OFAI acknowledges basic
financial support by the Austrian Federal Ministries for
Education, Science, and Culture and for Transport, Innovation, and Technology. We want to specially thank Simon
Dixon for essential help while programming these tools in
JAVA and for valuable comments on earlier versions of
this manuscript.

References
Dixon, S. (2001). Automatic extraction of tempo and beat fromexpressive performances. Journal of New Music Research, 30(1),39–58.
Goebl, W. (2001). Melody lead in piano performance: Expressivedevice or artifact? Journal of the Acoustical Society of America,
110(1), 563–572.
Hiraga, R., & Matsuda, N. (2004). Visualization of music performance as an aid to listener's comprehension, Proceedings ofthe Working Conference on Advanced Visual Interfaces (AVI) (pp.
103–106). Gallipoli, Italy: ACM Press.

5