Proceedings of the International Symposium on Musical Acoustics, March 31st to April 3rd 2004 (ISMA2004), Nara, JapanA convolutional-kernel based approach for note onset detectionin piano-solo audio signals
Gabriel Pablo Nava1, Hidehiko Tanaka1, Ichiro Ide2
1

Department of Information and Communication Eng., The University of Tokyo, Japan
2
SoftwareResearch Division, National Institute 2of Informatics, Japan1{pablo/tanaka}@mtl.t.u-tokyo.ac.jp

Abstract
This paper presents a new approach to piano onset
detection based on a convolutional kernel modeled fromthe salient characteristics of the transients of piano
onsets. Preliminary experiments performed with piano
recordings, show that our method promises to achievereasonable high level of onset detection even incomplex signals such as piano jazz and piano
improvisations with considerably fast tempi. Anotheradvantage of this approach is the viability to beextended to detect onsets of other instruments with few
modifications.

1. Introduction
Onset detection has been studied in computer music inorder to aid tasks such as automatic music transcription,
audio signals segmentation, tempo and beat tracking, etc.
These systems usually include in their first processing
stages an algorithm to discover in the raw signal, thereliable cues representing the real musical events thatcan be taken into account to infer higher levels of music
information. Since the detection of onsets is usually thestarting point in such systems, the subsequent processes
rely greatly on the information revealed by the onset
detector. Therefore, the implementation of an algorithm
that can find and label the true musical events, becomesa problem that deserves attention.
In this work, we concentrated on the detection ofnote onsets in recordings of piano-solo signals.
Researches on onset detection have been done in areasof beat and tempo tracking such as in [1][2][3]. The
onset information that these systems retrieve is that of
the beats defining the tempo and the rhythmic structure
for the signal analyzed and not precisely the location of
all the note onsets played in the original musicperformance. Thus, such systems are inappropriate for
automatic music transcription for example, where themain task is to transcribe the actual notes from the audio
signal. In recent approaches, [4] has presented a systemthat performs onset detection in the complex domain, incontrast with traditional onset detectors that consider
only the energy of the signal. The system of [4] works
well for onsets that concentrate in the lower frequency
spectral components, and onsets that present in some

ide@nii.ac.jp

degree hard attacks. Another approach for detecting
onsets of piano notes was proposed in [5], which isbased in a gammatone filterbank with output channels
centered at piano tones, and a smoothing filter thatworks together with a peak-picking step. This work wasimproved in [6] by employing two smoothing filterswith different time constant, in combination with aneural network. Although the smoothing filters giveconsiderable cues for onset inference, the neuralnetwork is unable to handle onsets with proximities lessthan 50 ms due to the period of inactivity of the neurons[6]. In [3], a particularization was done for onset of
piano signals and the approach was presented in [7],which integrates the basic technique of onset detection
in time domain [3] with analysis in frequency domain
and a genetic algorithm. Finally, [8] presented aprobabilistic model in which the problem of onset
detection is viewed as surprising moments that should
be observed and evaluated by a probability model using
independent component analysis.
In our approach, the idea is to analyze the signal in
the frequency domain by using a convolutional kernelthat will respond to the transient events whenever a
piano note is present. Piano sounds have theparticularity of presenting abrupt changes of energy inthe fundamental frequency components as well as in thenoisy components that conform the transient. This
characteristic hard attack in the noise components is dueto the fact that piano sounds are of percussive nature.
By finding an optimum kernel that can respond to thischaracteristic, we can convolve it with the spectral datain order to make evident the transient events. In Figure
1a, there is an example of the characteristic spectrogramof a piano key sound showing the characteristics
mentioned above. In the cases of more complex piano
signals (as in Figure 1b) the overlapping spectral
components make the accurate detection of onsets
difficult, leading to spurious detection and misses ofsome onsets. To solve this problem, the signal is divided
into sub-bands and a probabilistic network will evaluate
the hypotheses of possible true onsets detected.

2. System overview
The over all system for onset detection is shown in
Figure 2. The three main sections are: preprocessing,Figure 1.Eexample of spectrograms of piano sounds.
a) A single piano note, b) A piece of piano jazz fragment.

onset hypotheses labeling and the probabilistic network
for the evaluation. The input signal is antialiasing
filtered and then down-sampled from 44.1 kHz to 22kHz.
2.1. Preprocessing
In the cases of simple piano signals, as in Figure 1a, theappropriate kernel can be applied and the onsets can belocated easily. However, in real situations of piano
performances, many onsets cannot be detected due tothe masking effect of some spectral components into
others, as can be appreciated in Figure 1b where themagnitude of some harmonics are higher than other note
tones. This motives us to split the signal into sub-bands.
In this approach we use four sub-bands assigned asfollows:•Sub-band 1: (0 – 500 Hz)•Sub-band 2: (500 – 2500 Hz)•Sub-band 3: (2.5 – 5 kHz)•Sub-band 4: (5 – 10 kHz)
After filtering, the sub signals are down sampled,
then the spectrogram of each band is computed bysliding a Hanning window whose size and overlapping
are different for each sub-band and are selected so as toachieve the best resolution of time in the four bands (theachieved resolution in this model is 11.6 ms). The
resulting spectrogram is a three dimensional matrix withdimensions f, t and S(fi,tk), where:•f = f1, f2, f3, …, fM = frequency axis; M = number offrequency bins.
•t = t1, t2, t3, …, tN = discrete time index of the Nframes within a segment signal.
•S(fi, tk) = spectral power in the bin fi at moment tk; i =1,2,3 , …, M ; and k = 1,2,3, …,N.

Figure 2. Scheme of the onset detection system.
The absolute values of the spectral power are taken to
perform the convolution of the sub-band spectrogramwith its corresponding kernel. The kernels are matrices,of different sizes for each sub-band, and theircoefficients are computed from transients of piano
notes. The samples of piano notes used to train andconstruct the kernels were taken from the piano
recordings in [8]. For notes that are within thecorresponding sub-bands of our system, parameterssuch as attack time, maximum peak, rising slope andaverage separation of partials were measured and thevalues that minimize the least squared error in theconvolution are taken to form the kernel that will be
used for that sub-band.
With the corresponding kernel for each band, the twodimensional convolution with the magnitude spectral
power matrix is performed as follows:
mnhSB (i, j ) = ∑ ∑ S (k1 , k 2 ) K (i − k1 , j − k 2 )k1 =1 k 2 =1

where: K(i – k1, j – k2) is the kernel, m and n are therows and the columns of the kernel, and in this case, i =1, 2, . . ., M-m, and j = 1, 2, . . ., N-n.

The resulting convolved matrices hSB of the subbands are then normalized so that the magnitudes range
in the interval 0-1, and passed to the onset labelingstage.
2.2. Onset hypotheses labeling
Before starting the labeling of onsets, the normalized
magnitudes of the hSB matrices are summed along thefrequency axis f. This function generates a onedimensional signal with prominent peaks corresponding
to the possible onsets. Then, by peak-picking, a binary
signal is generated, indicating the presence or absenceof possible onsets that should be evaluated at theprobabilistic network. In Figure 3, an example of thespectrogram of a piano signal processed in the lowest
frequency sub-band is shown in a), together with theresulting convolution with the kernel in b), and the onedimensional signal of the energy integration function,c).
2.3. The probabilistic evaluation network
The train of pulses generated in the previous stage
represents labels where hypothetically an onset wasfound. However, the final decision about whether they
represent true onsets or not is taken based on aprobabilistic evaluation. From each hypothetic onset
signal of the sub-bands, the labels are observed and thedata are fed to the network nodes shown in Figure 4.

Figure 3. Example of a jazz piano 2 sec segmentprocessed at the lowest frequency analysis sub-band. a)
Signal spectrogram, b) Matrix after convolution,c) The prominent peak signal.
By using Bayes theorem, the probabilities can be
expressed as:
P(A) = P( Om | A) P(OM | A)
which corresponds to the probabilities computed by them and M evaluator after having observed the evidencesin the hypotheses labels.
The hypotheses with their corresponding
probabilities are then integrated at the onset
information integrator and the final labels for true
onsets are assigned.

3. Experiments

Figure 4. A single node for the probabilistic evaluationof the onset hypotheses.

Let us consider that we want to find the probabilityof the true onset inferred in A. Then letting Om thehypothesis of an onset detected, and OM be thehypothesis of the true onset detected and that will be
propagated to the next nodes. This can be written as:
P(A) = P(A | OM , Om)
where A is the vector of the hypothesis generated by thehypothesis generator, A=(a1, a2, a3, . . ., am).

We performed preliminary experiments for the overallsystem evaluation. First, the samples used to train andconstruct the kernels were single piano notes sounds
obtained from a standard music database [8], whichcontain sounds of some instruments recorded at asampling rate of 44.1 kHz and 16 bits of resolution.
The notes recorded were played with variations inarticulation (Normal, Stacatto, Pedal), and dynamics(Forte, Mezzo, Piano), summing a total of 30 notes thatwere analyzed to train each kernel. In the CD labeled as
RWC-MDB-I-2001-W01 of [8], the pictures of thepianos used to record the database are presented(showing a YAMAHA piano and a STEINWAY &SONS among them).
On the other hand, tests done until now, consist on
30 sec. segments of piano-solo performances recorded
at 44.1 kHz and 16 bits of resolution, taken fromdifferent CD’s and tracks of the same database.
Although
until now few classic and jazz piano

samples have been tested with our system, more
samples from commercial CD’s and in other styles arebeing considered to verify the performance of thesystem. Table 1 resumes the sample segments used forpreliminary tests.
The onsets of the sample segments were firstlabeled manually and then compared to those detectedby the system. Then we estimate the overall
performance as follows:

POD =Ncd − Ed× 100%
Tot

where:• POD = Percentage of correct onset detection.
• Ncd = number of onsets correctly detected.
• Ed = erroneous detections.
• Tot = number of the total actual onsets (labeledmanually for the corresponding segments).
These partial evaluations indicated average rates of
90.7% of onset detection for piano-solo in jazz style,and 94.3% for classic style, rates that are a meaningfulindicative of the viability for improvements on thismethod.
ClassicPiece# of segments tested
Mozart, “Piano Sonata inA major, K. 331/300i”
1Bethoven, “Piano Sonata
no. 23 in F major”
1Choppin, “Nocturne no. 2in Eb major”, “Etude in Emajor”
2
Total
4
JazzPiece# of segments tested
Jive (Piano solo)
1
For two (Piano solo)
1
Crescent Serenade (Pianosolo)
1
Total
3
Table 1. Sample segments used for evaluation.
In this first approach to piano onset detection byusing a convolutional kernel, we have modeled thekernel with parameters that defined at a first glance thetransients of the piano onsets. However, the system has
shown to be robust to spurious onset detections with theaid of the probability network. In further studies, thekernel can be refined to consider more complex
characteristics of the piano onset transients and withthis achieve higher levels of correct detection for mostof the piano performance styles. Another extension thatis considered for this method is the implementation of
new kernels modeled from onsets of other instruments,
so that the system will be theoretically able to detect
onsets not only from piano sounds but also from other
instruments in polyphonic signals.

4. Conclusions
We presented a new approach to piano onset detection
based on a convolutional kernel modeled from the
piano onset transients. In fact, the convolutional
method used here is a particularized version of animage processing technique used to detect edges ingray-scaled images. The particularization resides in thecore of this technique, the convolutional kernel. Thismethod has proved to be useful to aid onset detection
when applied to the analysis of the spectrogram ofaudio signals. Therefore, we can expect that, combining
and updating this approach with more advanced andefficient techniques from image processing and patternrecognition, better results can be achieved.

References[1] Scheirer E., “Tempo and beat analysis of acoustic
musical signals”, Journal of Acoustic Soc. ofAmerica, 103(1), 1998.
[2] Goto M., and Muraoka, “Real-time beat tracking
fordrumlessaudiosignals”,
SpeechCommunications, 27(3-4):331-335, 1999.
[3] Dixon S., Goebl M. and Widmer G., “Real-timetracking and visualization of musical expression”,Proceedings of the 2nd International Conference
ICMAI 2002, UK, 2002.
[4] Dubuxury C., Bello J., Davies M., and Sandler M.,“Complex domain onset detection for musicalsignals”, Proc. of the 6th Intl. Conf. on Digital
Audio Effects (DAFX-03), London, UK, 2003.
[5] Marolt M., “Adaptative oscillator networks forpartial tracking and piano music transcription”,
Proc. of the 2000 Intl. Computer Music Conf.,
Berlin, Germany, 2000.
[6] Marolt M., “SONIC: Transcription of polyphonicpiano music with neural networks”, Workshop on
CurrentRresearch Directions in Computer Music”,
Barcelona, Spain, 2001.
[7] Dixon S., “Learning to detect onsets of acoustic
piano tones”, MOSART Workshop on Current
Research Directions in Computer Music, Barcelona,Spain, 2001.
[8] RWC Music Database, 2003.03, Real World
Computer Partership, C MUSIC Corporation, Japan.