Music Transcription:
Identification of Musical Note Played on a Piano

Ayano Hiranaka
Mentored By Amirhossein Taghvaei
University of Illinois at Urbana-Champaign
June 2019

Contents
1. Introduction .................................................................................................................. 1
2. Model Generation and Visualization ........................................................................... 1
3. Feedback Particle Filter Algorithm ............................................................................. 2Algorithm 1: Calculate the Gain in FPF Algorithm ............................................................... 3
4. Empirical Studies on Performance of FPF Algorithm ................................................. 4
4.1 Effect of the Number of Oscillators N .................................................................... 5
4.2 Effect of Model Uncertainty.................................................................................... 5
4.3 Effect of Observation Model Uncertainty ............................................................... 5
5. Adaptive Filtering ........................................................................................................ 65.1 Coefficient Adapting ............................................................................................... 7
5.2 Noise Strength Adapting ........................................................................................ 76. Application on Real Signals ......................................................................................... 7
7. Note Classification ....................................................................................................... 98. Detecting Change in Note Played .............................................................................. 11
8.1 Reducing Reaction Time ....................................................................................... 11

1. Introduction
The main objective of this project is to identify, in real-time, the note being played on a piano.
An audio file of piano notes being played are inputted into an adaptive filtering algorithm
which compares the input signal to each note’s ideal signal models and predicts the note being
played through a probabilistic approach. The oscillator model is used to model the audio
signals, and the filter used is the coupled oscillator feedback filter.

2. Model Generation and Visualization
An oscillator model used to model the audio signals is developed. The ideal oscillator
position (angle in radians) θ at any instance in time k (up to K) and the observation signal Yk,
or the real audio signal with noise, are modeled as follows: is described as follows:
Ideal Oscillator:
Observation Signal:

𝜃𝑘+1 = (𝜃𝑘 + 𝜔∆𝑡) %2𝜋
𝑌𝑘 = ℎ(𝜃𝑘 ) + 𝜎𝑤 𝑊𝑘

(1)
(2)

Where ω is the signal frequency in rad/s and Δt is the time step size, ℎ(𝜃𝑘 ) is the observation
function, Wk is the normally distributed noise, 𝜎𝑤 is the noise strength (𝜎𝑤 ≥ 0), and θ isassumed to have an initial value of θo at k = 0 selected randomly from an uniform distribution[0, 2𝜋].
First, the observation function is assumed to take the following form where c is a constant:Observation Function:ℎ(𝜃𝑘 ) = 𝑐𝑠𝑖𝑛𝜃

Figures 1 and 2 showing plots of the ideal oscillator and the observation signal are obtained using
𝜔 = 2𝜋, ∆𝑡 = 0.01, 𝜎𝑤 = 0.1, 𝑐 = 1.0, 𝐾 = 500 , and shows the change in the ideal andobserved oscillator’s behavior with respect to time. The observation signal’s behavior issimilar to that of the ideal signal, and varies according to the normally distributed noise.
Figures 3 and 4 show the occurrence of θo and θ50 after the system is ran 𝑀 = 100 times.
The occurrences of θ values are generally uniform.

1

(a) Time Trace of Ideal Oscillator(b) Ideal and Observation Signals(c) Histogram of Occurrence of θo(d) Histogram of Occurrence of θ50

Figure 1: Behavior of the Oscillator Model and the Observation Signal

3. Feedback Particle Filter Algorithm
In this step, assume that only the observation signal is available (the ideal signal the observation signal
represents is unknown). The goal of this section is to predict θk given Yk with N oscillators. Each of

the N oscillators are initialized with a frequency 𝜔0𝑖 and a position 𝜃0𝑖 randomly selectedfrom uniform distributions with ranges [𝜔0 − 𝛿, 𝜔0 + 𝛿] and [0, 2𝜋] , respectively. TheFPF algorithm is applied to each of the N oscillators to predict the ideal oscillator theobservation signal represents. The oscillators evolve in the following way:2

FPF:𝑖𝜃𝑘+1= 𝜃𝑘𝑖 + 𝜔𝑖 ∆𝑡 + 𝐾 𝑖 (𝑌𝑘 −ℎ(𝜃𝑘𝑖 ) + ℎ̂ 1) 2 %2𝜋2𝜎𝑤(3)1𝑖𝑖Where 𝛿 = 0.1 , ℎ̂ = 𝑁 ∑𝑁
𝑖=1 ℎ(𝜃𝑘 ) and 𝐾 is the gain. The gain is calculated in thefollowing algorithm:Algorithm 1: Calculate the Gain in FPF Algorithm
𝑖 𝑁
Input: {𝜃 𝑖 }𝑁
𝑖=1 , {h(𝜃 )}𝑖=1Output: {𝐾 𝑖 }𝑁
𝑖=1𝑁ℎ̂ =1∑ ℎ(𝜃 𝑖 )𝑁𝑖=1
𝑁

1𝐴11 = ∑ 𝑠𝑖𝑛2 (𝜃 𝑖 )𝑁𝑖=1𝑁

1𝐴22 = ∑ 𝑐𝑜𝑠 2 (𝜃 𝑖 )𝑁𝑖=1

𝑁

1𝐴12 = 𝐴21 = − ∑ 𝑠𝑖𝑛(𝜃 𝑖 )𝑐𝑜𝑠(𝜃 𝑖 )
𝑁𝑖=1

𝑁

1
𝑏1 = ∑ 𝑐𝑜𝑠(𝜃 𝑖 )(ℎ(𝜃 𝑖 ) − ℎ̂)
𝑁𝑖=1
𝑁

1
𝑏2 = ∑ 𝑠𝑖𝑛(𝜃 𝑖 )(ℎ(𝜃 𝑖 ) − ℎ̂)
𝑁𝑖=1

A = [[𝐴11 , 𝐴12 ], [𝐴21 , 𝐴22 ]]b = [𝑏1 , 𝑏2 ]c = 𝐴−1 𝑏
𝐾 𝑖 = −𝑐1 sin(𝜃 𝑖 ) + 𝑐2 cos(𝜃 𝑖 ) 𝑓𝑜𝑟 𝑖 = 1, … , 𝑁
Figures 5 to 8 shows the results of the FPF algorithm. As more time elapses, the N oscillators3

converge to the true oscillator’s position. At k=0, the N oscillator’s prediction of the trueoscillator’s position is random, as shown in Figure 6. For larger values of k, such as 100, theprediction becomes more accurate, as shown by Figure 8.

(a) Time Trace of True and Predicted(b) Histogram of Distribution of Oscillators at

Oscillators

Time k=0(c) Histogram of Distribution of Oscillators at(d) Histogram of Distribution of Oscillators at

Time k=10

Time k=50
Figure 2: Results of the FPF Algorithm

4. Empirical Studies on Performance of FPF Algorithm
In this section, the performance of the FPF algorithm is investigated when: (1) the numberof oscillators N is varied, (2) there is uncertainty in the ideal oscillator model, and (3) there
is uncertainty in the observation model. In order to quantify the performance, the error of thealgorithm is defined in the following way:

4

𝑛−1

Error:

2

𝑁

𝑁

2

1
1
1
𝑒𝑛 = ∑ [( ∑ sin(𝜃𝑘𝑖 ) − sin(𝜃𝑘 )) + ( ∑ cos(𝜃𝑘𝑖 ) − cos(𝜃𝑘 )) ]𝑛𝑁𝑁𝑘=0𝑖=1(4)

𝑖=1

4.1 Effect of the Number of Oscillators N
The effect of the number of oscillators is investigated by plotting the error with respect to thenumber of oscillators n for N = {10, 20, 50, 100, 200, 500, 1000}. The result is shown in
Figure 3a. The error was expected to decrease as the number of oscillators used are increased,
but Figure 3a shows an interesting result suggesting that there is little difference in error
resulting from a change in the number of oscillators used.
4.2 Effect of Model Uncertainty
Next, the effect of uncertainty in the ideal signal model is investigated. Practically,parameters of the ideal signal model will most likely be unknown, for example, the true
frequency of the signal, ω, may be unknown. In the algorithm, the N oscillators’ frequencies
𝜔𝑖 are selected from the range [𝜔0 − 𝛿, 𝜔0 + 𝛿] . It is predicted that the FPF filter willfunction properly when the true frequency of the input signal is within this range. This section
investigates the FPF algorithm’s performance when the true frequency is outside of this range.
For this purpose, the ideal oscillator frequency is defined as ω = 2π(1 + α) and 𝛿 = 0.2,
and the error values are compared for α = {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6}. Results are shownin Figure 3b. As expected, the error is higher for higher values of α, or when the actual
frequency deviates more from the expected range.
4.3 Effect of Observation Model Uncertainty
For this analysis, the observation function is assumed to take the form:ℎ(𝜃) = 𝑐1 𝑠𝑖𝑛𝜃 + 𝑐2 𝑐𝑜𝑠𝜃.
To investigate the effect of observation model uncertainty on error, the FPF algorithm will
be applied to the oscillators assuming that the observation function takes the form as initiallydefined:ℎ(𝜃𝑘 ) = 𝑐0 𝑠𝑖𝑛𝜃.
Let𝑐0 = 1.0 and C = [(𝑐1 , 𝑐2 )] = [(1.5, 0.0), (0.5, 0.0), (1.0, 0.5)] . The resulting error isshown in Figure 3c. (𝑐1 , 𝑐2 ) = (1.5, 0.0) results in the highest error while (𝑐1 , 𝑐2 ) =(0.5, 0.0) results in the lowest error, suggesting that the error becomes high when the5observation model coefficient exceeds the coefficient of the true model. The performance
when (𝑐1 , 𝑐2 ) = (1.0, 0.5) suggests that the error is not affected significantly when theactual signal contains a harmonics not assumed by the algorithm (𝑐2 𝑐𝑜𝑠𝜃) if the weight ofthis component is sufficiently low.

(a) Time Progression of Error for Different
Number of Oscillators N(b) Effect of Model Frequency Uncertaintyin Error Progression(c) Effect of Observation Model
Uncertainty on Error Progression
Figure 3: Results of Empirical Studies on FPF Performance

5. Adaptive Filtering
Results of section 4.3 reveals that the FPF algorithm’s performance is sensitive to how well
the assumed observation model matches the actual observed signal.

65.1 Coefficient AdaptingThis section considers a way to adapt the coefficients of the assumed observation function tofit the actual observation function. To better accommodate various observation functions,assume the function takes the following form:
Observation Function:ℎ(𝜃) = 𝑐1 𝑠𝑖𝑛𝜃 + 𝑐2 sin(2𝜃) + 𝑐3 cos (2𝜃)(5)

In every iteration of the FPF algorithm, the coefficients 𝑐1 , 𝑐2 , and 𝑐3 evolves in thefollowing way for 𝑛 = 1, 2, 3:𝑁𝜀𝑎𝑚𝑝 𝜕1(𝑌𝑘 − ∑ ℎ(𝜃𝑘𝑖 ))𝑐𝑛 ↩ 𝑐𝑛 − 2𝜎𝑤 𝜕𝑐𝑛𝑁⏟
𝑖=12(6)𝑒𝑟𝑟𝑜𝑟

Where 𝜀𝑎𝑚𝑝 = 0.1 is the amplitude correction factor. In this approach, the coefficients areadjusted along the direction of the error gradient, reducing the error each time the coefficientsare updated.
5.2 Noise Strength Adapting
The performance of the FPF algorithm is sensitive to the error strength 𝜎𝑤 . The noise levelmay be unknown or varying, and will be adapted in the following way:𝜎𝑤2 ↩ (1 − 𝜀𝑛𝑜𝑖𝑠𝑒 )𝜎𝑤2 + 𝜀(𝑌𝑘 − ℎ̂𝑘 )2
Where 𝜀𝑛𝑜𝑖𝑠𝑒 ≪ 1 is the noise correction factor.

(7)

6. Application on Real Signals
In this section, the FPF algorithm is applied to a real piano note signal. Audio files of single
piano keys played once are inputted to the algorithm and its responses are observed. Thenotes selected as inputs are {𝐶6, 𝐷6, 𝐸6, 𝐹6, 𝐺6, 𝐴6, 𝐵6, 𝐶7}. Prior to entering signals into
the system, each signal was investigated and reasonable starting values for the observation
model coefficients (𝑐1 , 𝑐2 , 𝑐3 ) and the noise strength 𝜎𝑤2 are determined for each notes
signal. The results are shown in Figure 4. It is observed that the algorithm is able to trace
each note accurately.

7

(a) C6(b) D6(c) E6(d) F6(e) G6(f) A68

(g) B6(h) C7

Figure 4: Extracted Portions of Time Traces of True Input Signals and FPF Predicted Signals{𝐶6, 𝐷6, 𝐸6, 𝐹6, 𝐺6, 𝐴6, 𝐵6, 𝐶7}

7. Note Classification
The objective of this section is to identify which note is being played given one of the notesfrom {𝐶6, 𝐷6, 𝐸6, 𝐹6, 𝐺6, 𝐴6, 𝐵6, 𝐶7} as input. The FPF algorithm is applied to the input
signal eight times, each time assuming that the input signal is one of the eight notes: First,
the algorithm is applied assuming that the input note is C6, the next iteration of the algorithm
is applied assuming that the input note is D6. This process is repeated until all eight notes areconsidered. Each time, the prediction oscillator attempts to trace the input signal, and theprediction signal that is able to trace the input signal most accurately is identified as the note
(1)

(2)

(8)

played. To quantify how well the input is predicted, the filter exponents {𝜇𝑘 , 𝜇𝑘 , … , 𝜇𝑘 }are initialized at zero and update according to:Filter Exponents:

(𝑚)

(𝑚)

𝜇𝑘+1 = 𝜇𝑘−

12𝜎𝑤2
⏟

2(𝑚)
(𝑦𝑘 − ℎ̂𝑘 )
(𝑚)

(8)

𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑜𝑛 𝑒𝑟𝑟𝑜𝑟(1) (2)(8)
Where {ℎ̂𝑘 , ℎ̂𝑘 , … , ℎ̂𝑘 } are prediction outputs for each of the eight notes, obtained byaveraging the values of the N oscillators at each time step k. Results when each of the eightnotes are inputted is shown in Figure 5. Notes {𝐶6, 𝐷6, 𝐸6, 𝐹6, 𝐺6, 𝐴6, 𝐵6} are identified
correctly. The difference in the exponents of E6 and F6 are small when the input note is F6,and C7 is incorrectly identified as B6, suggesting that the algorithm may not be able to9reliably distinguish between notes that are a half-note apart.

(a) C6 as Input(b) D6 as Input(c) E6 as Input(d) F6 as Input(e) G6 as Input(f) A6 as Input

10(g) B6 as Input(h) C7 as Input

Figure 5: Exponents for Inputs {𝐶6, 𝐷6, 𝐸6, 𝐹6, 𝐺6, 𝐴6, 𝐵6, 𝐶7}

8. Detecting Change in Note Played
In order to transcribe a musical piece played on a piano, the algorithm must be able to detecta change in the note being played. The objective of this section is to identify a change in thenote being played in real-time using the exponents values.
8.1 Reducing Reaction TimeWhen, for instance, the note C6 is played, the exponent for C6 shows the highest value among
the eight exponents. Then, when the note changes, for example, to G6, the exponent of G6
begins to increase. However, the system will not identify the new note G6 until the exponent
for G6 surpasses the exponent of C6. In order to reduce this delay, a threshold is implemented,
preventing exponent values to decrease below a certain value. While a higher thresholdincreases reaction speed, it also increases the likelihood of error. An optimum level of thethreshold is found using a portion of a piano recording of the “ABC Song” where the notechanges from C6 to G6. To allow the ideal samples to better fit the input signal, the input
song signal’s amplitude is reduced to 150. Results for 𝜀𝑎𝑚𝑝 = 0.01 , 𝜀𝑛𝑜𝑖𝑠𝑒 = 10−5 ,𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 = −106 is shown in Figure 6. The note played changes from C6 to G6 at the redvertical line, and the system identifies this change after a delay of approximately 0.263
seconds.

11

Figure 6: Change in Exponent as the Note Played Changes from C6 to G6

12