11th International Society for Music Information Retrieval Conference (ISMIR 2010)

QUERY-BY-CONDUCTING: AN INTERFACE TO RETRIEVE
CLASSICAL-MUSIC INTERPRETATIONS BY REAL-TIME TEMPO INPUT
Akira Maezawa,† Masataka Goto‡ and Hiroshi G. Okuno††

Dept. of Intelligence Science and Technology
Graduate School of Informatics, Kyoto University
Sakyo-ku, Kyoto 606-8501 Japan{amaezaw1, okuno}@kuis.kyoto-u.ac.jp‡

National Institute of Advanced Industrial
Science and Technology (AIST)
Tsukuba, Ibaraki 305-8568 Japanm.goto@aist.go.jp

ABSTRACTThis paper presents an interface for finding interpretationsof a user-specified music, Query-by-Conducting. In classical music, there are many interpretations to a particularpiece, and finding “the” interpretation that matches the listener’s taste allows a listener to further enjoy the piece. The
critical issue in finding such an interpretation is the way
or interface to allow the listener to listen through different interpretations. Our interface allows a user, by swinging a conducting hardware interface, to conduct the desired
global tempo along the playback of a piece, at any time in
the piece. The real-time conducting input by the user dynamically switches the interpretation being played back to
the one closest to how the user is currently conducting. At
the end of the piece, our interface ranks each interpretationaccording to how close the tempo of each interpretation
was to the user input.
At the core of our interface is an automated tempo estimation method based on audio-score alignment. We improve tempo estimation by requiring the audio-score alignment of different interpretations to be consistent with eachother. We evaluate the tempo estimation method using asolo, chamber, and orchestral repertoire. The proposed
tempo estimation decreases the error by as much as 0.94
times the original error.

Figure 1: System diagram of Query-by-Conducting.

not made or distributed for profit or commercial advantage and that copies

Our goal is to retrieve interpretations1 on the basis ofvarious aspects of music interpretation, similar to contentbased music information retrieval (CBMIR), which retrieves pieces on the basis of various aspects of music suchas rhythm and timbre [2, 3]. What constitutes musical interpretation is a difficult question, though it seems that musicians express interpretation by manipulating the tempo,
the timbre, or bringing out interesting melodic lines. This
paper focuses on the global tempo – the average tempo
of a piece over a few beats. We believe tempo is an aspect of music interpretation that many listeners take noteof. Studies in music cognition suggest that similarity of interpretations is strongly reflected in global tempo [4], andmany studies are motivated by the significance of tempo
on interpretation [1, 5].
We present Query-by-Conducting, a new interface forfinding interpretations of a user-specified music by conducting the global tempo. The interface reads, as the music score, a standard MIDI file of a piece of music, anddifferent interpretations of the music score as audio files.
The interface facilitates playback, visualization and queryof interpretations by supporting the following features, as
shown in Figure 1:
1. Visualization of global tempi of the interpretations
2. Hardware conducting interface (a Nintendo Wii remote) for intuitively entering, along the playback ofa piece, the user’s tempo query in real time
3. Ranking and retrieval of interpretations on the basisof the similarity between each interpretation and thecurrent tempo query entered by the user.
Visualization allows the user to view the range of interpretations available, and thus, the valid range of tempo

bear this notice and the full citation on the first page.
c 2010 International Society for Music Information Retrieval.
⃝

1 We shall use the term “interpretation” to mean a rendition of a particular symbolic representation of music, as per [1].

1. INTRODUCTIONClassical music is unique in that many audio recordings
exist for a given piece of music. For example, as ofMarch 2010, a search on an on-line shopping site for“Mendelssohn Violin Concerto” returns 1200+ hits, or thatof “Beethoven Spring Sonata” returns 300+ hits. Eachof these recordings is an acoustic rendition of a particular music score, embodied by an unique interpretation of
the performer. Finding an interpretation that matches thelistener’s taste is an important aspect of enjoying classicalmusic. However, searching for such recording is tiresome
because it requires the listener to listen through the same
piece many times.
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are477

11th International Society for Music Information Retrieval Conference (ISMIR 2010)

Figure 2: System-level state diagram. The interface alternates
between (1) and (2) until the piece ends.

Figure 3: The interface in playback-mode visualizes the tempo
along playback of an interpretation.

in which the user should conduct to obtain a meaningful
query. As a particular interpretation is played back, theuser may become dissatisfied with its tempo. Then, theuser would “conduct” the desired tempo using the hardware conducting interface. Based on the user’s tempo input, the interface retrieves and switches the interpretation
being played back to one that is closest to the input. The
user may stop conducting if the current interpretation issatisfactory. At the end of playback, the system ranks eachinterpretation on the basis of how similar the overall global
tempo trajectory was to the user’s conducting.
Our interface differs from existing conducting interfaces [6–9] in three respects. First, we use conducting device to switch the interpretation being played back, insteadof specifying the tempo of the entire piece. The user has
the freedom of either listening to a piece, or conducting
the tempo of an interpretation that the user wants to listen. Second, our method allows the user to control only
the global tempo instead of local tempo or dynamics. We
believe that such restriction is an effective way to retrievea particular interpretation; global tempo is easy for a typical user to specify, but specifying local tempo requires
a precise control of the tempo. The notion of restricting
the user control to a few dimensions has been proposedin other studies aimed at easily manipulating expressive
music [10]. Finally, our conducting interface is meant toretrieve a particular interpretation to play back, whereas
most conducting interfaces are aimed at real-time temporal manipulation of a particular audio signal. Unlike queryby tapping [11], which uses rhythm pattern as the query,as our method uses the tempo as the query.
The interface relies on tempo estimation that is determined through audio-score alignment. In existing studies [1, 12–14], audio-score alignment was created using
only the information obtained from the audio of interestand the score. There may be errors in the alignment, but
given one alignment, there is no way of knowing wherean error is. When aligning multiple interpretations, however, it is also possible to create audio-score alignment byaligning the score to some other audio, and then aligningthat audio to the audio of interest. Thus, given N interpretations to align, N unique audio-score alignments to
one interpretation can be generated. We use these multiple audio-score alignments generated to estimate the true
audio-score alignment that is error-free.

A video demonstration of our interface is available at
http://www.youtube.com/QueryByConducting

2. INTERFACE DESIGN
The interface offers functions in a conventional music
playback interface such as playback and rewind. Moreover, it features a visualizer of global tempi of various interpretations, a conducting hardware to enter the global
tempo query in real time, and retrieval of interpretations
on the basis of the user query. As shown in the state diagram in Figure 2, our system alternates between playing
back the current interpretation (“playback-mode”), and accepting user conducting and retrieving appropriate interpretation to play back (“conduct-mode”). At the end, oursystem ranks each interpretation on the basis of the tempo(“ranking-mode”).
Figure 3 shows the interface during playback(playback-mode). Bottom of the screen displays thetitle, the performer and the playback time, similar toconventional music playback interface. Top of the screen
visualizes the global tempi, and presents each interpretation sorted in descending order of the current global
tempo. Bottom right shows the state of the conducting
interface.
As the piece is played back, the user may become dissatisfied with the tempo of the piece (“I liked the introduction, but the development section is too slow,” a user might
think). As shown in Figure 4, the interface allows a user
to “conduct” the desired global tempo in conduct-mode,in real time. In conduct-mode, the interface accepts beat
input from the conducting hardware interface, and also visualizes the beat at the bottom left to facilitate proper conducting. The entered tempo is used as a query to retrieve
the interpretation whose global tempo is closest to what
the user conducts, and to switch the current playback to
it. This mode offers the user an active listening experienceby constantly retrieving and cross-fading the playback tointerpretation that plays like how the user is conducting.
At the end of the piece, the interface enters the rankingmode, and ranks each interpretation based on how similar
each interpretation was to the user’s overall conducting.
The ranking is presented to the user.

478

11th International Society for Music Information Retrieval Conference (ISMIR 2010)

Figure 5: Interpretation visualizer shows the tempo trajectoryof the current interpretation, how the user has conducted, and therange of tempi.
Figure 4: The interface in conduct-mode accepts user’s con-

2.2 Hardware Conducting Interface

ducting using a controller, and switches interpretations being
played back. Beat visualizer facilitates user’s conducting.

The hardware conducting interface detects beat from anaccelerometer embedded in the hardware controller, andconverts it into tempo. The user interface shows beat visualizer to facilitate tempo entry.

2.1 Interpretation Visualizer
Top half of the interface (in Figure 3) is the interpretation
visualizer. It shows the tempi of different interpretations,along with tempo trajectory of the interpretation that is being played back and the user query.
Figure 5 shows the visualizer in further detail. It
presents tempo information against time. To allow the user
to view the detailed tempo near the current playback position as well as the tempo of the entire piece, we distort thenormalized x-coordinate, x(t). x(t) is distorted such thatthe vicinity of current playback position tc is zoomed likea lens. Let t, the current beat of playback, be defined for[0, tl ], where tl is the duration of the piece, and the rangeof x ∈ [0, 1]. Then, the visualizer applies the following
function:` t−t ´1 exp T ct
` t−tc ´+x(t) =
2tl
2 exp T + 1

2.2.1 Beat detection
We accept the user’s conducting query using a game controller that features a 3-axis accelerometer (a Nintendo Wii
controller). Our system detects beat by checking for peaksin the axis vertical to the controller. Such peak is generated when the controller is flicked up, as a conductor would
flick the baton to indicate the beat.
Once a beat is detected, the accelerometer input is ignored for 200msec to prevent false triggering. Therefore,
our system accepts tempo of up to 300 beats-per-minute(BPM), which is sufficient for virtually all classical music.
2.2.2 Converting beat input to tempo query
To specify a new tempo, the user must conduct a tempo different from the playback. We observed that people tends
to conduct not in the desired tempo, but instead ahead or
behind of the beat of the playback to indicate faster orslower tempo relative to the current playback. We conjectured that such phenomenon occurs because people are
distracted by the downbeat of the playback, and sets the desired beat location relative to the last downbeat he/she hasheard. Therefore, we convert the offset of the user’s conducting with respect to the beat of the playback, to the desired tempo in BPM. Suppose the user conducts ∆t behindthe beat. Then, supposing the current BPM of the playback
is BP M0 , we convert ∆t to user-specified tempo, BP M ,as follows:(1)

This shows approximately T -neighborhood of the current
playback position in more detail than the rest. T is chosen
to be four quarter notes.
A vertical straight line indicates the current playback
position. Line segment to the left of the current playback
position is the past tempo trajectory of the user’s global
tempo query. Line segment to the right of the current playback position is the future tempo trajectory of the interpretation that is being played back. This way, the user isable to view the query entered so far, and how the current
interpretation will unfold.
To show the range of possible interpretations, the rangeof global tempi is expressed as a colorful strip, overlayed
to to the line segments described above. The strip is colored using a gradation of hue angle, such that fast tempo isassociated with small hue (orange), and slow tempo withlarge hue (blue). At beat t, given the slowest tempo τmin (t),fastest tempo τmax (t), and some tempo in between, τ (t),we set the hue to the following angle:
τ (t) − τmin (t)hue(t) = 240 −230◦τmax (t) − τmin (t)◦BP M = BP M0

1∆t+1
60/BP M0(3)

The average of user-specified BPM over four beats is usedas the query.
2.2.3 Beat visualizer
We observed that, in a preliminary experiment using a few
test subjects, people did not always have a clear sense ofrhythm, and had trouble finding where the beat is. Thiswas especially true for music whose instrumentation did
not include instrument with strong attack and decay, suchas the piano or plucked strings.
To facilitate tempo input, we display a beat visualizer,
as shown at the bottom left of Figure 3, and in detail in(2)

Right half of the visualizer prints the performer of eachinterpretation, sorted in descending order of the current
global tempo. The interpretation that is being played backis highlighted. Next to each name, a box whose hue value
is as described in Equation (2) is painted.

479

11th International Society for Music Information Retrieval Conference (ISMIR 2010)

Figure 6: Visualizer for facilitating tempo input. The color bar
rotates in synchrony with the beat so that the user could easily
grasp the downbeat.

Figure 7: Ranking-mode ranks each interpretation based on
the tempo similarity, presents similar interpretations as opaqueand dissimilar ones transparent.

Figure 6. The visualizer has a colored stripe that rotatesaround the large circle, in synchrony with the beat of theplayback. At each downbeat, the stripe crosses the small
circle at the bottom. The arc-length of the rotating stripe
corresponds to the range of global tempi at the current beat,and the hue is calculated using Equation (2). Therefore, if
the user wants to switch to fast interpretation, for example, from tempo shown as green in the tempo visualizer toorange-colored tempo, the user could flick the controller
as the orange-colored segment of the arc crosses the small
circle at the bottom.

interpretation. Let cS be a 12 dimensional chroma vector computed from the music score at tick t. We generate
the alignment from the score to the kth interpretation, denoted Mk←s , or from interpretation i to j, denoted Mj←i .
To generate Mk←s , a similarity matrix Rk←s is first computed. Let Ns be the number of ticks in the music scoreand Nk be the number of audio frames contained in interpretation k. Let Rk←s be a Ns -by-Nk matrix, whose
element i, j contains:(i)(j)

2.3 Interpretation Retriever

Next, we find the alignment path using DTW. Formally, wedefine a cost matrix Ns -by-Nk matrix,
as follows:
8(t)

Rk←s (i, j) = 1 −After the user has finished listening through a piece, theinterface ranks, in ranking-mode, each interpretation on
the basis of the similarity between the tempo trajectory of
the interpretation and the user query.
We use the tempo trajectories of the interpretations thatwere played as the query. For example, if the user listened
to interpretation x for the first minute and y for the next two
minutes, our query would consist of the tempo trajectory ofinterpretation x for the first minute and y for the next two.
Let us define the dissimilarity score of the user queryand each interpretation. Let τi (t) be the global tempo trajectory of the ith interpretation, and τq (t) be the query
tempo trajectory. Then, we define tempo dissimilarity forinterpretation i, ri as follows:
„
«
ri =

1
T

Z T
0

τi (t) − τq (t)τq (t)

Ck←s (i, j) = Rk←s (i, j) + min><Ck←s (i − 1, j)
Ck←s (i, j − 1)>:Ck←s (i − 1, j − 1)

(5)

(6)

where for all t, Ck←s (t, −1) = Ck←s (−1, t) = 0. Next,
we determine the parametric representation of the audio(t)score alignment, Mk←s , by backtracking the cost matrix.
(0)First, we set Mk←s to (Ns , Nk ), and update in the follow(t)ing manner while incrementing t, until Mk←s = (0, 0):
(t+1)

Mk←s

:=

S

=

argmin Ck←s (I, J)(7)(I,J)∈S{(i − 1, j), (i, j − 1), (i − 1, j − 1)}(t)where (i, j) = Mk←s . Audio-audio alignment from interpretation i to j, Mj←i , can be achieved in the same way,by computing the similarity matrix between chroma vector
sequence of interpretation i and j.
3.2 Improving Audio-Score Alignment
We improve audio-score alignment by requiring the alignments of different interpretations to be consistent witheach other. Given one music score and N interpretations,
there are N possible paths to generate the alignment fromthe music score to interpretation i, as shown in Figure 8.
Namely, in addition to the direct mapping from the score
to interpretation i, it is also possible to generate mappingfrom the score to interpretation j (audio-score alignment),
which is then mapped by using the map from interpretation
j to i (audio-audio alignment). Ideally, all N paths fromthe score to an interpretation should be identical. In reality, however, they are not because they are generated using
different similarity matrices.
In order to generate a map from the score to some interpretation i via interpretation j, Mi←j ◦Mj←S , both Mj←S
and Mi←j must be one-to-one, but the alignments generated in the previous section are not.
Therefore, we trace, over the alignment determined in
the previous section, a new map that is one-to-one. We perform the following procedure for each alignment between
some interpretation (or score) s and k:

2dt

cs · ck(i)(j)∥cs ∥ · ∥ck ∥(4)

The interpretations are sorted in the ascending order of
tempo dissimilarity, as shown in Figure 7. A transparency
value is associated to each interpretation being drawn, suchthat interpretation with lowest dissimilarity is opaque, andthe highest transparent. Moreover, dissimilarity measurethat is inverted, shifted and scaled between 0 and 1 isshown next to each interpretation.
3. TEMPO EXTRACTION METHOD
Global tempo extraction is based on evaluating the audioscore alignment. Since accurate alignment is essential for
accurate tempo estimation, we propose a method to improve the audio-score alignment.
3.1 Initial Audio-Score Alignment
The initial audio-score alignment is based on dynamic
time-warping (DTW) using chroma vector as the feature,similar to other works [13–15].
(t)
Let ck be a 12 dimensional vector that contains thechroma vector computed for tth audio frame of the kth

480

11th International Society for Music Information Retrieval Conference (ISMIR 2010)
Table 1: Average MSE (mean-squared error) improvementin thousandths (10−3 ) after iterating Equation (12).
Piece (No. Interp.)
solo-1 (13)solo-2 (6)solo-3 (5)duo-1 (5)duo-2 (4)duo-3 (4)orch-1 (5)orch-2 (5)orch-3 (5)Figure 8: By combining two alignments, there are multiple ways to align the score to an interpretation.
1. Set t = 0, and the initial point of the refined alignment(t)M̃k←s to (0, 0).

n

(8)

where
(n)
(t)
d(t)
n (q, θ) = ∥M̃k←s + q (cos(θ), sin(θ)) − Mk←s ∥ (9)Q is chosen to be 20 frames, and ϵ to be π/20 radian. θ isevaluated every π/20 radians.
3. Update M̃k←s as follows, for some ∆r ∈ (0, 1]:
“
”
(t+1)
(t)
M̃k←s
:= M̃k←s + ∆r cos(θ̂), sin(θ̂) (10)θ̂=argmin c(θ)
θ

MSE =

We chose ∆r = 1 frame.
(t)

5. Set t := t + 1, and go to 2.

We assume that observed alignments are corrupted byindependent and identically distributed noise that follows
the Laplace distribution with location parameter M̂i←S (t)and scale parameter b, for each beat t:(11)

and likewise for Mi←j ◦ Mj←S for j ̸= i. We interpret
M̂i←S as the underlying “correct” alignment that generates
Mi←S and Mi←j ◦ Mj←S . Since an estimator of M̂i←S is
the sample median, we update Mi←S as follows:
Mi←S (t) := median({Mi←j ◦ Mj←S (t)})(12)

As will be shown in the experiment, iterating this stepyields in improved alignment accuracy.
3.3 Tempo Extraction
The tempo is estimated by determining the slope of theaudio-score alignment. We compute the tempo at MIDI
tick t using alignment information obtained between tickt − T to t + T for T > 0. Only information at note onsets
are used, as alignment results between two note onsets are
not reliable. We choose T dynamically such that at least 20
audio frames that correspond to note onsets are within thisrange. Let (s(p), a(p)) contain a parametric representationof Mi←S that contain the audio frames chosen. s corresponds to the domain (tick of note onsets) and a the range(audio frame). Then, we compute the BPM at tick t, τ (t)by first finding the slope m(t) of (s(p), a(p)) using linear
regression, and multiplying its inverse by a scalar factor:
τ (t) =

1 audio frame-per-minutem(t)ticks-per-beatIter. 2
8.4
13.0
85.4
3.7
20.6
10.2
47.2
13.3
1038.3Iter. 10
8.4
12.7
98.8
3.8
20.4
10.2
44.9
13.2
833.2

1
T

T

0

τg (t) − τ̂ (t)τg (t)

2

dt

(14)

MSE can be considered as the dissimilarity measure between the ground truth and the estimated tempo.
Table 1 shows the average of MSE over all interpretation for each of the nine pieces, as the number of iterationsof the update step (Equation (12)) is changed.
The results suggest that, first, our method is capable ofdecreasing the error, more so if the initial error is high. For
example, duo-3 has its error decreased by 0.94 times theoriginal error, after ten iterations. Second, in most cases,
iterating our method multiple times yields in decreased error. When the error increases with increased number of
iterations, we believe that our assumption that alignments
are corrupted by independent noise fails. For example, in
pieces that involve unnotated candenza (e.g. solo-3), incorrect alignment occurs consistently at the cadenza. Then,
taking the median of such corrupted data yields not in
the underlying “true” alignment, as our method posits, butsome meaningless data instead.
4.2 Evaluation of Music Query
We evaluate the robustness of our system against errors in
conducting. When a user conducts like some interpretationi, the system should retrieve i as the most similar interpretation. Other results may be returned for two reasons:
1. The user could not conduct the piece accurately
enough to return the desired query.
2. Imprecision in tempo estimation method causes incorrect result to be returned.
In these cases, i may not be the most similar, but one of M
most similar interpretations.
First, we synthesize an artificial query that models human errors in conducting, by adding a smooth noise to theground truth tempo trajectory of each data. For each interpretation i, we use the following tempo trajectory as thequery with some noise variance s :(t)

4. Exit if M̃k←s · (1, 0) ≥ Ns or M̃k←s · (0, 1) ≥ Nk .

“”p(t) = exp −∥Mi←S (t) − M̂i←S (t)∥/b /2bIter. 1
8.6
15.0
73.1
3.9
22.1
12.5
54.4
14.7
1091.4nine pieces, three are orchestral (denoted orch-1 to orch3), three are written for small ensemble (denoted duo-1 toduo-3), and three are solo piano (denoted solo-1 to solo-3).
For each work, multiple interpretations (between four andthirteen) were obtained and their ground truth tempo data
were entered using an in-house tempo entry utility.
4.1 Evaluation of Audio-Score Alignment
Let τg (t) be the ground truth tempo trajectory. Given an
estimated tempo trajectory τ̂ (t), we evaluate the error using scaled mean squared
error (MSE), defined
as follows:«
Z „2. For ϵ < θ < π2 − ϵ, compute the following cost function:c(θ) = Eq∼exp (−3q/Q) [min d(t)n (q, θ)]

None
8.9
17.3
266.7
4.5
34.8
185.4
646.8
231.5
3941.6(13)

4. EXPERIMENTS
We evaluate the tempo estimation method, and retrieval ofinterpretation on the basis of global tempo query. We analyzed nine classical pieces of varying instrumentation. Of

τquery,i (t; s)n(t; s)481=∼

τg,i (t) · 2
N (0, 1)

√ s PL−1L

l=0n(t−l)|L = 10 (15)

11th International Society for Music Information Retrieval Conference (ISMIR 2010)

F-measure

1
0.8

s=0.0
s=0.1
s=0.15
s=0.2
s=0.5

0.6
0.4
0.2
0

fers the listener an interactive experience of “conducting”
the global tempo to dynamically tailor the interpretation
played back to the user’s choice. It moreover presents thelistener with a ranking of interpretation based on how theuser conducted through the piece, offering the listener witha list of interpretations whose tempi that the user might
like, without the hassle of listening through various interpretations. The accuracy of tempo estimation method improved as a result of considering the consistency of audioscore alignment among different interpretations.
As future work, we would like to deal with aspectsof music interpretation other than the global tempo, suchas the local tempo deviation and emphasis of a particular melodic line. Integrating these aspects would further
enhance the system’s capability to retrieve the interpretation of choice. Furthermore, we would like to realize more
ways to visualize and interact with various aspects of music interpretation, to allow a listener to further enjoy classical music.
Acknowledgment: This research was partially supported
by Graint-in-Aid for Scientific Research (S) of the Ministry of Education, Culture, Sports, Scince and Technology(MEXT), and the CrestMuse Project of the Japan Scienceand Technology Agency (JST).

Noise variance

Top 1

Top 2

Top 3

Top 4

Top 5

(a) solo-1
F-measure1
0.8

Noise variance
s=0.0
s=0.1
s=0.15
s=0.2
s=0.5

0.6
0.4
0.2
0

Top 1

Top 2

Top 3

Top 4

Top 5

(b) duo-1
F-measure1
0.8

Noise variance
s=0.0
s=0.1
s=0.15
s=0.2
s=0.5

0.6
0.4
0.2
0

Top 1

Top 2

Top 3

Top 4

Top 5

(c) orch-1
Figure 9: Evaluation results when retrieving up to top 5 resultsthat are similar to artificially generated query which deviates fromthe ground truth by variance s.

Next, we retrieve M interpretations that are most similar to the artificial query for each interpretation, and evaluate the performance of retrieval the F-measure. Let nbe the number of interpretations correctly retrieved by thequery. Let N be the total number of interpretations. Then,
we let recall R = n/N , and precision P = n/(N × M ).
The F-measure F is 2P R/(P + R). We show the resultsfrom solo-1, duo-1, and orch-1 in Figure 9 (a), (b), and (c).
Figure 9 (a) and (b) show that the system tolerates small
error in conducting, of up to about s = 0.15, or 0.7 to 1.3
times the original tempo (three-sigma). Figure 9 (c), however, shows that the F-measure of orch-1 is considerably
lower than the other two.
Retrieving orchestral piece (orch’s) is difficult because
there is very small variation in the two closest playing, andexacerbated by the particularly unreliable tempo estimation. We compute the smallest dissimilarity measure between ground truth tempo trajectories of any pair of interpretations. The smallest dissimilarity of orch-1 is about
2 × 10−3 , solo-1 is 20 × 10−3 , and duo-1 is 23 × 10−3 . We
similarly observed that for orchestral piece, the smallest
dissimilarity is much smaller compared to that of chamber (duo’s) or solo (solo’s). On the other hand, we observethat the average MSE, as seen in Table 1, is substantiallygreater for orchestral pieces than chamber or solo.
These results suggest that our system retrieves the desired interpretation with robustness against minor errors in
conducting, as long as the average MSE is small enough todifferentiate the most similar pair of interpretations. The
similarity of interpretation is typically influenced by thescale of orchestration, and the average MSE is influenced
by the complexity of the ensemble, and the degree to which
the interpretation deviates from the music score.
5. CONCLUSION
This paper presented Query-by-Conducting, an interface
for finding interpretations of a given piece of music. It of-

6. REFERENCES[1] M. Müller et al. Towards automated extraction of tempo parameters from expressive music recordings. In ISMIR ’09,
pages 69–74, 2009.
[2] C.A.Michael et al. Content-based music information retrieval: Current directions and future challenges. volume 96,
pages 668–696, April 2008.
[3] S. Dixon, F. Guyon and G. Widmer. Towards characterizationof music via rhythmic patterns. In ISMIR ’04, pages 509–516,
2004.
[4] R. Timmers. Predicting the similarity between expressive
performances of music from measurements of tempo and dynamics. JASA, 117(1):391–399, 2005.
[5] H. Honing. From time to time: The representation of timing
and tempo. Comp. Music J., 25(3):50–61, 2001.
[6] S.Schertenleib et al. Conducting a virtual orchestra. IEEE
MultiMedia, 11(3):40–49, 2004.
[7] J. Segen, S. Kumar and J. Gluckman. Visual interface for
conducting virtual orchestra. ICPR ’00, page 1276, 2000.
[8] H. Katayose and K. Okudaira. Using an expressive performance template in a music conducting interface. In NIME
’04, pages 124–129, 2004.
[9] F.Bevilacqua et al. Wireless sensor interface and gesturefollower for music pedagogy. In NIME ’07, pages 124–129,
2007.
[10] S. Dixon, W. Goebl and G. Widmer. The ”air worm”: Aninterface for real-time manipulation of expressive music performance. In ICMC ’05, pages 614–617, 2005.
[11] J.R.Jang, H.Lee and C.Yeh. Query by tapping: A newparadigm for content-based music retrieval from acoustic input. In PCM ’01, pages 590–597. Springer-Verlag, 2001.
[12] F. Kurth et al. SyncPlayer - an advanced system for multimodal music access. In ISMIR ’05, pages 381–388, 2005.
[13] S.Dixon and G.Widmer. MATCH: Music alignment tool
chest. In ISMIR’05, pages 11–15, 2005.
[14] N.Hu, R. Dannenberg and G. Tzanetakis. Polyphonic audio
matching and alignment for music retrieval. In WASPAA ’03,
pages 185–188, 2003.
[15] M. Müller, F. Kurth and T. Röder. Towards an efficient algorithm for automatic score-to-audio synchronization. In ISMIR’04, pages 365–372, 2004.

482