ICASSP 2003 (The 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing) Proceedings, pp.V-437-440, April
2003.

A CHORUS-SECTION DETECTING METHOD FOR MUSICAL AUDIO SIGNALS
Masataka Goto
“Information and Human Activity,” PRESTO, Japan Science and Technology Corporation (JST). /National Institute of Advanced Industrial Science and Technology (AIST).
1-1-1 Umezono, Tsukuba, Ibaraki 305-8568, JAPAN. < m.goto@aist.go.jp >ABSTRACTThis paper describes a method for obtaining a list of chorus (refrain) sections in compact-disc recordings of popular music. The
detection of chorus sections is essential for the computational modeling of music understanding and is useful in various applications,such as automatic chorus-preview functions in music browsers or
retrieval systems. Most previous methods detected as a chorus a
repeated section of a given length and had difficulty in identifying
both ends of a chorus section and in dealing with modulations (keychanges). By analyzing relationships between various repeatedsections, our method called RefraiD can detect all the chorus sections in a song and estimate both ends of each section. It can also
detect modulated chorus sections by introducing a similarity thatenables modulated repetition to be judged correctly. Experimental
results with a popular-music database show that this method detects the correct chorus sections in 80 of 100 songs.
1. INTRODUCTION
Chorus (refrain) sections of popular music are the most representative and prominent thematic sections in the music structure of asong, and human listeners can easily understand where the chorus
sections are because these sections are most repeated and memorable portions of a song. Their automatic detection is essential forbuilding a computational model that can understand musical audio signals in a human-like fashion, and is useful in various practical applications. In music browsers or music retrieval systems,
it enables a user to quickly preview a chorus section as an “audio
thumbnail” in order to find a desired song. It can also increase
the efficiency and precision of music retrieval systems by enablingthem to match a query with only the chorus sections.
Most previous chorus-detection methods [1, 2, 3] obtained asingle segment from several chorus sections by detecting a repeatedsection of a given length as the most representative of a song. Logan and Chu [1] developed a method using clustering techniquesand Hidden Markov Models to categorize short segments (1 sec)
in terms of their acoustic features and then regarded the most frequent category as a chorus. Bartsch and Wakefield [2] developed
a method that calculated the similarity between acoustic featuresof beat-length segments obtained by beat tracking and found thegiven-length segment with the highest similarity averaged over its
segment. Cooper and Foote [3] developed a method that calculated the similarity between acoustic features of short frames (100ms) and found the given-length segment with the highest similarity between it and the whole song. None of the previous methods,
however, addressed the problem of detecting all the chorus sections in a song. They also assumed that the output segment length
is given and did not identify both ends of a chorus section. Whilechorus sections are sometimes modulated (the key is changed) during their repetition in a song, the previous methods were not able
to deal with modulated repetition.
This paper describes a method, called RefraiD (Refrain
Detecting Method), that exhaustively detects all the chorus sections appearing in a song. It can obtain a list of the start and end

c0-7803-7663-3/03/$17.00 2003
IEEE

points of every chorus section in real-world audio signals and can
detect modulated chorus sections. Furthermore, because it detects
chorus sections by analyzing various repeated sections in a song,it can generate an intermediate-result list of repeated sections thatusually reflect the music structure of the song; for example, therepetition of the structure like the verse A, verse B, and chorus isoften found in the list.
The following sections describe the problems dealt with, specify the RefraiD method in detail, and show experimental results
indicating that the method is robust enough to detect the correct
chorus sections in 80 of 100 songs of a popular-music database.
2. CHORUS-SECTION DETECTING PROBLEM
Given an audio signal of a song, we want to obtain a list of all thechorus sections without using any prior information about the spectral characteristics of chorus sections. Because the chorus sections
are usually the most repeated sections in popular music, the basic idea behind dealing with this problem is to find various groupsof repeated sections and then output the group that appears mostfrequently. It is, however, generally difficult to find the repeatedsections automatically because they do not completely match eachother. The main issues can be summarized as follows:[Problem 1] Acoustic features and similarity measure
Whether or not one section is the repetition of another must be
judged on the basis of the similarity between acoustic features of
those sections. The simple power spectrum or MFCC features
are not powerful tools for this judgment because they are liable
to change considerably when arrangements of accompaniments
or melody lines are changed after repetition.
[Problem 2] Repetition-judgment criterion
The appropriate criterion of the similarity for judging repetition depends on the song. For a song containing many-repeatedaccompaniment phrases, for example, only a section with very
high similarity should be considered the chorus-section repetition. For a song containing a chorus section with accompaniments changed after repetition, on the other hand, a section withsomewhat lower similarity can be considered the chorus-section
repetition. Because a criterion hand-tuned for a few songs is notrobust for a large song set, the criterion must be adjusted automatically for each song.
[Problem 3] Estimating both ends of repeated sections
It is necessary to estimate the start and end points of repeatedsections by analyzing relationships between various repeatedsections. For a song whose structure is (A B C B C C), for example, the repetition of (B C) would be obtained by a simple
repetition search. In this case, both ends of this C can be estimated by using the information of the repetition of the last C.
[Problem 4] Detecting the modulated repetition
Although the modulated repetition is difficult to detect because
acoustic features are not similar after a modulation, this detection is important because the repetition of chorus sections (especially around the end of song) sometimes includes the modulation.

V - 437ICASSP 2003



Audio signal

Calculating the similarity between chroma vectors
Listing the repeated sections
Integrating the repeated sections
Integrating the modulated repeated sections

Considering 12 kinds ofthe pitch difference bymodulations (key changes)

Extracting 12-dimensional chroma vectors

Selecting the chorus sections
List of chorus sections

List of repeated sections

Fig. 1. Overview of the chorus-section detecting method RefraiD.

3.2. Calculating the similarity

3. CHORUS-SECTION DETECTING METHOD:REFRAID
Figure 1 shows an overview of the RefraiD method. It first extracts
a 12-dimensional feature vector called a chroma vector, which isrobust for small changes of accompaniments, from each frame of
an input audio signal and calculates the similarity between these
vectors (solution to Problem 1). Each element of the chroma vector corresponds to one of the 12 pitch classes and is the sum of
power at frequencies of its pitch class over six octaves. The method
then lists pairs of repeated sections by using an adaptive repetitionjudgment criterion that is configured by an automatic threshold selection method based on a discriminant criterion [4] (solution to
Problem 2). To organize common repeated sections into groups
and to identify both ends of each section, it integrates those pairs by
analyzing their relationships over the whole song (solution to Problem 3). Because each element of a chroma vector corresponds to adifferent pitch class, a before-modulation chroma vector is close to
the after-modulation chorus vector whose elements are shifted (exchanged) by the pitch difference of the key change. By considering
twelve kinds of shifts (pitch differences), it then calculates 12 setsof the similarity between non-shifted and shifted chroma vectors,
lists pairs of repeated sections from those sets, and integrates all ofthem (solution to Problem 4). Finally, it evaluates the possibilityof being chorus sections for each group and outputs the repeatedsections with the highest possibility as well as other groups of repeated sections.
3.1. Extracting acoustic features
The 12-dimensional chroma vector v(t) is extracted from the power
spectrum, Ψp (f, t) at the log-scale frequency f at time t, calculated
by using the short-time Fourier transform (STFT). Each elementof v(t) corresponds to a pitch class c (c = 1, 2, . . . 12) in the equal
temperament and is represented as vc (t):vc (t) =OctH  ∞

h=OctLBPFc,h (f ) Ψp (f, t) df.



2π(f − (Fc,h − 100))1
1 − cos. (3)2200This filter is applied to octaves from OctL to OctH .
In the current implementation, the input signal is digitized at
16 bit / 16 kHz, and then the STFT with a 4096-sample Hanningwindow is calculated by using the Fast Fourier Transform (FFT).
Since the FFT frame is shifted by 1280 samples, the discrete time
step (1 frame shift) is 80 ms. The OctL and OctH , the octave range
for the summation of Equation (1), are respectively 3 and 8. This
covers six octaves (130 Hz to 8 kHz).
There are several advantages to the chroma vector.3 Because
it captures the overall harmony (pitch-class distribution), it canbe similar even if accompaniments or melody lines are changed
in some degree after repetition. In fact, we have confirmed thatthe chroma vector is effective for identifying chord names. The
chroma vector also enables the modulated repetition to be detected
as described in Section 3.5.
BPFc,h (f ) =

(1)

−∞

The BPFc,h (f ) is a bandpass filter that passes at the log-scale frequency Fc,h (in cents1 ) of pitch class c in octave position h2
Fc,h = 1200h + 100(c − 1)(2)
and is defined using a Hanning window as follows:
1 Frequency f
Hz in hertz is converted to frequency fcent in cents so that
there are 100 cents to a tempered semitone and 1200 to an octave: fcent =31200 log2 (fHz / (440 × 2 12 −5 )).
2 In the Shepard’s helix representation of pitch perception [5], c and hrespectively correspond to chroma and height.

The similarity r(t, l) between the chroma vectors v(t) and v(t − l) v(t)
is defined as− maxvc(t−l)
maxc vc (t)vc (t−l)√,(4)r(t, l) = 1 −12√where l (0 ≤ l ≤ t) is the lag. Since the denominator 12 is thelength of the diagonal line of the 12-dimensional hypercube withedge length 1, r(t, l) satisfies 0 ≤ r(t, l) ≤ 1.
3.3. Listing the repeated sections
Pairs of repeated sections are obtained from r(t, l). Consideringthat r(t, l) is drawn within the right-angled isosceles triangle in thetwo-dimensional time-lag space as shown in Figure 2, the methodfinds line segments that are parallel to the horizontal time axis andindicate consecutive regions with high r(t, l). When the section
between the time T 1 and T 2 is denoted [T 1, T 2], each line segment between the points (T 1, L1) and (T 2, L1) is represented as(t = [T 1, T 2], l = L1) and means that the section [T 1, T 2] is similar to (i.e., is the repetition of) the section [T 1 − L1, T 2 − L1].
In other words, a line segment indicates a repeated-section pair.
To find (t = [T 1, T 2], l = L1) in r(t, l), the possibility ofcontaining line segments at the lag l, Rall (t, l), is evaluated at thecurrent time t (e.g., at the end of song) as follows (Figure 2): tr(τ, l)dτ.
(5)Rall (t, l) =t−l
lBefore this calculation, r(t, l) is normalized by subtracting themean of r(t, l) in the adjacent area while removing noises. The
method then picks up high peaks above a threshold ThR ofRall (t, l) for searching the line segments, after smoothing Rall (t, l)
by using a moving average filter. Because the threshold ThR isclosely related to the repetition-judgment criterion that should be
adjusted for each song, we use an automatic threshold selection
method based on a discriminant criterion [4]. When dichotomizing
the peak heights into two classes by a threshold, the optimal threshold is obtained by maximizing the discriminant criterion measurethat is defined by the following between-class variance:2= ω1 ω2 (µ1 − µ2 )2 ,(6)
σB
where ω1 and ω2 are the probabilities of class occurrence (the number of peaks in each class / the total number of peaks), and µ1 andµ2 are the mean of peak heights in each class.
The line segments are finally searched in the direction of
the horizontal time axis on the one-dimensional function r(τ, L1)(L1 ≤ τ ≤ t) at the lag L1 of each high peak. After smoothing
3 The chroma vector is similar to the chroma spectrum [6] that is usedin reference [2], although its formulation is different.

V - 438

3.5. Detecting the modulated repetitionl (lag)l (lag)Line segments corresponding
to the repetition of the half of C
Line segments corresponding
to the repetition of C
Line segment corresponding
to the repetition of ABCC
Line segment
corresponding to therepetition of C

A: verse A
B: verse B
C: chorus

0r (t,l)
A

B

C C

A

B

C C C C t (time) Rall (t,l)

Fig. 2. A plot of line segments, the similarity r(t, l), and the possibility Rall (t, l) of containing line segments. The similarity r(t, l)is defined in the right-angled isosceles triangle in the lower righthand corner. The actual r(t, l) is noisy and ambiguous and usuallycontains many line segments irrelevant to chorus sections.
r(τ, L1) by using a moving average filter, the method obtains line
segments on which the smoothed r(τ, L1) is above a threshold.
This threshold is also adjusted by using the automatic threshold
selection method.
3.4. Integrating the repeated sections
Since each line segment indicates just a pair of repeated sections,
it is necessary to organize into a group the line segments that havecommon sections. First, line segments that have almost the same
section [Tsi , Tei ] are organized into a group, which is represented
as φi = ([Tsi , Tei ], Γi ), where Γi = {γij | j = 1, 2, . . . , Mi } (Miis the number of line segments) is a set of the segment’s lag γij —corresponding to the high peaks in Rall (t, l) — in this group. A
set of these groups is denoted by Φ = {φi | i = 1, 2, . . . , N } (N isthe number of all the groups).
By using r(t, l) just within [Tsi , Tei ] of each group φi , line segments are then searched again in order to recover some line segments that the process described in Section 3.3 did not find. In
Figure 2, for example, we can expect that two line segments corresponding to the repetition of the first and third C and the repetition of the second and fourth C, which overlap with the longline segment corresponding to the repetition of ABCC, are foundhere even if they were hard to be found in the process described in
Section 3.3. For this purpose, starting from Teir(τ, l)dτ(7)Tei − TsiTsiinstead of Rall (t, l), the method performs the almost same peakpicking process described in Section 3.3 and forms a new set Γi ofthe peaks γij in R[Tsi ,Tei ] (l). It then removes inappropriate peaks
in each Γi as follows: it removes too many peaks that are equally
spaced, a peak whose line segment has a highly deviated r(τ, γij )(Tsi ≤ τ ≤ Tei ), and a peak that is too close to other peaks andmakes sections overlap.
Finally, by using the lag γij corresponding to each peak of Γi ,the method searches for a group whose section is [Tsi − γij , Tei −γij ] (i.e., is shared by the current group Γi ) and integrates it with
Γi if it is found. They are integrated by adding all the peaks of thefound group to Γi after adjusting the lag values (peak positions);
the found group is then removed. In addition, if there is a groupthat has a peak indicating the section [Tsi − γij , Tei − γij ], it too
is integrated.
R[Tsi ,Tei ] (l) =The processes described above do not deal with the modulation(key change), but they can easily be extended to it. A modulation can be represented by the pitch difference of its key change,tr (0, 1, . . . , 11), which denotes the number of tempered semitones.
For example, tr = 9 means the modulation of nine semitones upward or the modulation of three semitones downward.
One of the advantages of the 12-dimensional chroma vector
v(t) is that tr of the modulation can naturally correspond to theamount by which its 12 elements are shifted. When v(t) is thechroma vector of a performance and v(t) is the chroma vector of
the performance that is modulated by tr semitones upward from theoriginal performance, they satisfy
v(t) =.. S trv(t) ,(8)
where S is a shift matrix defined by

 0



S =

0
.
.
.
0
0
1

1
0

0
1
..
.

···
0
..
.

···
···
0

···
···
···

0
···
···

···
···
..
.
1
0
···

0
0
.
.
.
0
1
0




.


(9)

To detect the modulated repetition, we can define 12 kinds ofextended similarity for each tr as follows:
rtr (t, l) = 1 − Str v(t) maxc vc (t) − maxvc(t−l)
vc (t−l) 
√

.
(10)
12
Starting from each rtr (t, l), the processes of listing and integrating the repeated sections are performed as described in
Sections 3.3 and 3.4, except that the threshold adjusted at tr = 0
is used for the processes at tr = 0. After these processes, 12 setsof line-segment groups are obtained for 12 kinds of tr. To organize non-modulated and modulated repeated sections into the samegroups, the method integrates several groups across all the sets ifthey share the same section.
Hereafter, we use Φ = {φi |φi = ([Tsi , Tei ], Γi )} to denote thegroups of line segments obtained from all the tr. By unfolding eachline segment of γij to the pair of repeated sections indicated by it,
we can obtain
Λi = {([Psij , Peij ], λij ) | j = 1, 2, . . . , Mi + 1},(11)
where [Psij , Peij ] = [Tsi − γij , Tei − γij ] represents one of theunfolded repeated section, and λij is its possibility of being chorus
sections. The λij is defined as the mean of the similarity rtr (t, l)on the corresponding line segment. For j = Mi + 1, we define[Psij , Peij ] and λij as follows: [Psij , Peij ] = [Tsi , Tei ] and λij =imaxM
k=1 λik . The modulated sections are labeled with their tr for
reference.
3.6. Selecting the chorus sections
After evaluating the total possibility νi of being chorus sections foreach group (φi , Λi ), the group m that maximizes νi is selected as
the chorus sections: m = argmaxi νi . The total possibility νi is asum of λij weighted by the length of the section and is defined by



Mi +1

νi =

j=1

λij

log

Tei − Tsi
,
Dlen

(12)

where Dlen is a constant (1.4 sec). Before calculating νi , the possibility λij of each repeated section is adjusted according to thefollowing three assumptions (heuristics):[Assumption 1] The length of the chorus section has an appropriate range (in the current implementation, 7.7 to 40 sec). If thelength is out of the range, λij is set to 0.
[Assumption 2] When there is a repeated section that is longenough to be likely to correspond to the long-term repetition likethe verse A, verse B, and chorus, the chorus section is likely to

V - 439

Table 1. Results of evaluating RefraiD: the number of songs whose
chorus sections were detected correctly under 4 sets of conditions.
Modulation detection
Use of assumptions 2 & 3
Number of songs (out of 100)
Fig. 3. The detected chorus sections of the database song RWCMDB-P-2001 No. 18. The horizontal axis is the time (sec) corresponding to the whole song. The window above shows the power.
The top line in the window below shows the list of the detectedchorus sections, which were correct for this song and the last ofwhich was modulated. The bottom five lines show the list of various repeated sections.
be near its end. If there is a repeated section [Psij , Peij ] whose
end is close to the end of another long repeated section (longer
then 50 sec), its λij is doubled.
[Assumption 3] Because a chorus section tends to have two halflength repeated sub-sections within its section, a section that has
those sub-sections is likely to be the chorus section. If there
is a repeated section [Psij , Peij ] that has those sub-sections inanother group, half of the mean of the possibility of those two
sub-sections is added to its λij .
These assumptions fit a large class of popular music.
4. EXPERIMENTAL RESULTS
The RefraiD method has been implemented in a real-time systemthat takes a musical audio signal as input and outputs the list of
the detected chorus sections. Along the real-time audio input, thesystem displays visualized lists of chorus sections and other repeated sections, which are obtained by using just the past input andare considered most probable every moment.4 A chorus-section
viewer that shows those visualized lists as shown in Figure 3 andenables a user to play back a selected section has also been developed.
The system was tested on 100 songs of the popular-music database “RWC Music Database: Popular Music” (RWC-MDB-P2001 No. 1−100) [7], which is an original database available toresearchers around the world. These 100 songs were originallycomposed, arranged, performed, and recorded in a way that reflected the complexity and diversity of real-world music. We compared the system output with the correct chorus sections that were
hand-labeled by using a music-structure labeling editor. The degree of matching between the detected and correct chorus sections
was evaluated by using the F-measure [8], which is the harmonicmean of the recall rate (R) and the precision rate (P ):2RP(13)
F-measure =R+P
sum of the length of chorus sections detected correctlyR=(14)
sum of the length of correct chorus sections
sum of the length of chorus sections detected correctly. (15)P =sum of the length of chorus sections detected
The system output of a song was judged to be correct if its Fmeasure is more than 0.75 under the condition that estimated trof modulations must be correct.
The results are listed in Table 1. The method dealt correctlywith 80 of 100 songs (the average of the F-measure of the 80 songs
was 0.938). The main reasons that it made mistakes were that the
number of repetition of chorus sections was not more than that of
4 Further information, including video clips, is available at the following
URL: http://staff.aist.go.jp/m.goto/ICASSP2003/

Condition (enable:◦, disable:×)

◦
◦

×

◦

×

◦

×
×

80

74

72

68

other sections and that an accompaniment phrase was repeated formost of a song. Among the 100 songs were 10 songs with modulated chorus sections, and the outputs of 9 of them were correct.
When the function detecting the modulated repetition was disabled,
only 74 songs were dealt with correctly. On the other hand, whenassumptions 2 and 3 were not used, the performance fell as shownby the entries in the rightmost two columns of Table 1. There were
22 songs in which accompaniments or melody lines of a repeatedchorus section were considerably changed; the outputs of 21 ofthem were correct and the repeated chorus section itself (i.e., tr)
was correctly detected in 16 songs.
5. CONCLUSION
We have described the RefraiD method that detects the chorus sections in real-world popular-music audio signals. It basically regards the most repeated sections as the chorus sections. Analysisof the relationships between various repeated sections enables all
the chorus sections to be detected with their start and end points.
In addition, the introduction of the similarity between non-shifted
and shifted chroma vectors makes it possible to detect modulated
chorus sections, which previous methods could not detect. Experimental results show that the method is robust enough to detect thecorrect chorus sections in 80 of 100 songs.
The RefraiD method also has relevance to music summarization methods [9, 10, 11], none of which addressed the problem ofdetecting all the chorus sections. One of the chorus sections detected by our method can be regarded as a song summary, as could
another long repeated section in the intermediate-result list of repeated sections.
Our repetition-based approach was proven effective in popular
music. To improve the performance of the method, however, we
will need to use prior information about the spectral characteristics
of chorus sections. We also plan to experiment with other music
genres and extend the method to be widely applicable.
ACKNOWLEDGMENTS: I thank Hideki Asoh for his valuable
discussions.
6. REFERENCES[1] B. Logan and S. Chu, “Music summarization using key phrases,” inProc. of ICASSP 2000, pp. II–749–752, 2000.
[2] M. A. Bartsch and G. H. Wakefield, “To catch a chorus: Using
chroma-based representations for audio thumbnailing,” in Proc. of
WASPAA’01, pp. 15–18, 2001.
[3] M. Cooper and J. Foote, “Automatic music summarization via similarity analysis,” in Proc. of ISMIR 2002, pp. 81–85, 2002.
[4] N. Otsu, “A threshold selection method from gray-level histograms,”
IEEE Trans. SMC, vol. SMC-9, no. 1, pp. 62–66, 1979.
[5] R. N. Shepard, “Circularity in judgments of relative pitch,” J. Acoust.
Soc. Am., vol. 36, no. 12, pp. 2346–2353, 1964.
[6] G. H. Wakefield, “Mathematical representation of joint time-chroma
distributions,” SPIE’99, pp. 637–645, 1999.
[7] M. Goto et al., “RWC music database: Popular, classical, and jazzmusic databases,” in Proc. of ISMIR 2002, pp. 287–288, 2002.
[8] C. J. van Rijsbergen, Information Retrieval. Butterworths, second ed., 1979.
[9] G. Peeters et al., “Toward automatic music audio summary generation from signal analysis,” in Proc. of ISMIR 2002, pp. 94–100, 2002.
[10] R. B. Dannenberg and N. Hu, “Pattern discovery techniques for music audio,” in Proc. of ISMIR 2002, pp. 63–70, 2002.
[11] K. Hirata and S. Matsuda, “Interactive music summarization basedon GTTM,” in Proc. of ISMIR 2002, pp. 86–93, 2002.

V - 440