A Novel Approach to
Separation of Musical Signal Sources by NMF
Sakurako Yazawa

Masatoshi Hamanaka

Takehito Utsuro

Graduate School of Systems
and Information Engineering,
University of Tsukuba, Japan

Department of Clinical
System Onco-Informatics,
Kyoto University, Japan

Faculty of Engineering,
Information and Systems,
University of Tsukuba, Japan

Abstract—This paper proposes a method to separate polyphonic music signal into signals of each musical instrument by
NMF: Non-negative Matrix Factorization based on preservationof spectrum envelope. Sound source separation is taken as a fundamental issue in music signal processing and NMF is becoming
common to solve it because of its versatility and compatibility
with music signal processing. Our method bases on a common
feature of harmonic signal: spectrum envelopes of musical signal
in close pitches played by the harmonic music instrument would
be similar. We estimate power spectrums of each instrument by
NMF with restriction to synchronize spectrum envelope of bases
which are allocated to all possible center frequencies of each
instrument. This manipulation means separation of components
which refers to tones of each instrument and realizes both of
separation without pre-training and separation of signal including
harmonic and non-harmonic sound. We had an experiment todecompose mixture sound signal of MIDI instruments into each
instrument and evaluated the result by SNR of single MIDI
instrument sound signals and separated signals. As a result, SNR
of lead guitar and drums approximately marked 3.6 and 6.0 dB
and showed significance of our method.
Keywords—NMF, spectrum envelope preservation, unsupervised
learning, polyphonic music signal separation,I.

I NTRODUCTION

The goal of our work is to extract signals of a specificmusical instrument from musical piece signal data. If it becomes possible to highly precisely extract sound sources ofany specific musical instrument, then one is able to extract
and operate musical parts and musical piece. It is further
expected that it enables certain operation of each musical part’s
volume, which then makes much higher level operation suchas arrangement of musical piece much easier than before.
We focus on a sound source of rock band formation andpropose a sound source separation technique for a sound sourceof rock band formation. In this paper, we assume that the soundsource of rock band formation is that of four part formation,
which consists of the lead guitar, the backing guitar, the base,and the drums. With this sound source, the sound signal is amusic sound signal of performance sound which is a mixtureof harmonic instruments and non-harmonic instruments. This
paper proposes how to separate musical sound signal of thistype into the sound signal of each musical instrument.
The performance of the rock band formation is popularamong music fans of young generation, who often like toimitate the performance of certain famous rock bands. Throughimitating the performance of certain famous rock bands andthe nuance of their performance, it is necessary for those
beginners, not only to simply follow note information, but tolearn how to vary strength in musical sound, to add certain
expression, and to change melodies from part to part bychanging performance. However, it is usually difficult for
beginners to distinguish the sound of a specific part from thesound source with more than one electric guitars for example.
Thus, those musical beginners definitely have strong needs fora technique of separating a sound source of specific parts.
Recently, in the field of the sound source separation, Nonnegative Matrix Factorization (NMF) has been intensively
studied. NMF is a technique that interprets the spectrogram ofthe sound signal as a non-negative matrix, and approximate it
as the product of a base vector group and a temporal activationgroup. NMF is often employed in certain formalization ofmusic sound signals for the following two reasons: (1) it can
express musical instruments through certain characteristics,and (2) additivity of the power spectrum of musical instruments
approximately holds. So far, there exist several previous workon the sound source separation using NMF including the followings: an approach of utilizing the musical score information
as the supervision signal [1], that of modeling harmonic andnon-harmonic structures and the reverberation information,
aiming at sound source separation and replacement of an
instrument performance phrase [2], that of modeling timevarying spectral patterns of musical instruments through NMF
framework [3].
However, those previous work are limited in that althoughsome of them satisfy just one of the following two requirements, any single technique by itself does not satisfy bothof the following two requirements: namely, (i) unsupervised
method of sound source separation without supervision information, (ii) realizing separation of harmonic and non-harmonic
instruments. As for the first requirement, supposing the casewhere one wants to separate the sound source of a musicalpiece that are unknown to him/her, it is common that he/she is
not given information such as musical scores in advance. As
for the second requirement, the fundamental technique of thesound source separation by NMF identifies temporal activationof a base vector group, while it lacks information on whichmusical instrument each base actually expresses.
In order to overcome those two obstacles, this paper
proposed a novel sound separation method based on NMF. Our
method bases on a common feature of harmonic signal: spectrum envelopes of musical signal in close pitches played by
the harmonic music instrument would be similar. We estimate

Fig. 1.

Proposed Model of Separating Musical Signal Sources by NMF

power spectrums of each instrument by NMF with restriction
to synchronize spectrum envelope of bases which are allocatedto all possible center frequencies of each instrument. This
manipulation means separation of components which refers
to tones of each instrument and realizes both of separation
without pre-training and separation of signal including harmonic and non-harmonic sound. We have an experiment todecompose mixture sound signal of MIDI instruments into
each instrument and evaluate the result by SNR of single MIDI
instrument sound signals and separate signals. As a result, SNR
of lead guitar and drums approximately mark 3.6 and 6.0 dB
and show significance of our method.
II.

R ELATED W ORK

Some previous work on sound source separation for mixed
sound source of more than one instruments models theacoustic characteristics of each musical instrument and takes
a parametric approach of updating and optimizing certain
parameters through EM algorithm. Among them, Goto [4]proposed PreFEst which estimates a melody line and a baseline from the mixed sound source of more than one musicalinstruments. Itoyama [5] defined a model of harmonics andnon-harmonics structure, and by utilizing the MIDI soundsource as a supervision information, proposed a techniqueof iteratively converging the model towards the target signal
through EM algorithm.
Among those work on sound source separation using NMF,
Uchiyama [6] proposed how to separate the sound source ofharmonic and non-harmonic instruments based on the fact
that, in spectrum, harmonic instruments tend to be smoothin temporal direction and to change drastically in frequency
direction, while non-harmonic instruments tend to be smoothin frequency direction and to change drastically in temporal
direction. This fact is also advantageous in our work for the
purpose of separating harmonic and non-harmonic mixture ofsound signal. However, this approach has limitation when the
mixture of sound signal contains more than one harmonic
instruments. Furthermore, even harmonic musical instruments
contain non-harmonic sound such as attack sounds of theguitar, while even non-harmonic musical instruments containharmonic sound such as the interval of the tom. Thus, it is notenough to simply separate harmonic and non-harmonic mixtureof sound signal, but it is necessary to invent a model which
can express both harmonic and non-harmonic sound of every
such musical instrument.
Another approach [7] utilized a base for each pitch nameand initialized them as a comb shape in the frequency space sothat they have harmonic structure, where they proposed how to
detect notes with chromatic scales. Although this approach isalso effective in our task of modeling separation of harmonic
instruments, it again has limitation when the mixture of soundsignal contains more than one harmonic instruments.
Schmidt [8] studied how to separate sound source of morethan one harmonic musical instruments under the following two assumptions: (a) harmonic structures of instruments’sound are shift immutable in logarithmic frequency, (b) the
power ratio of the harmonic overtone is fixed even if the central
frequency changes. This approach considers tones of harmonic
musical instruments as characteristics when separating thesound sources, while it is limited in that the tone of one
musical instrument is fixed for all the pitches. This assumption
is not appropriate in our work since musical instruments having
several octaves range have quite different tones according tohigh and low pitches. Thus, it is required in our work to model
continuously changing tones according to the height of pitches
in the framework of music signal source separation.
III.

T HE P ROPOSED A PPROACH

This sections roughly describes our approach to musicsignal source separation for the mixture of harmonic andnon-harmonic musical instruments with the performance inrock band formation. The musical instruments used in theperformance in rock band formation is mostly fixed to someextent, and is in general with the guitar, the base, the drum, andthe keyboard. However, the tone of each musical instrument
greatly varies according to musical pieces. In the case of theguitar, for example, their exist roughly two types of sound, one
is clean sound without any sound effect units, while the otheris those with certain distortion through sound effect units. Its

sound further changes if it is through a certain equalizer. Thissituation of the guitar is quite different from those studied in
the music source separation of other musical instruments suchas piano, the wind instruments, and the violin. In the case of
the guitar, it is difficult to provide the model with the acoustic
characteristics in advance, which makes it difficult to apply thesupervised learning approach. In order to solve this problem,in our proposed framework, we assume that the numbers andthe types of each musical instrument are given beforehand, andthen formalize bases of NMF under this restriction in order tolearn the tone of each musical instrument and then to realize
music source separation.
Figure 1 shows the structure of bases. The number ofbases varies according to each musical instrument and we
specify a certain number for each musical instrument. For each
non-harmonic musical instrument, we prepare two bases, oneof which is for non-harmonic sound, while the other is forharmonic sound accompanying the non-harmonic sound. For
each harmonic musical instrument, we prepare as many basesas the frequency bins included in all the frequency bands thatthe harmonic musical instrument has. Here, note that it is not
sufficient if one just prepares exactly the same number of bases
corresponding to the chromatic scales but no more bases. Thisis because it is possible to play harmonic musical instruments
such as the guitar with a high level playing technique suchas bending (after twanging a string, pulling and pushing thestring with fingers and changing the pitch smoothly), which
results in continuous changes of the pitch.
IV.

T HE M ODEL OF S EPARATING M USICAL S OUNDS OURCES

This section describes the details of the model of separatingmusical sound sources. In NMF, the iteration continues while
minimizing divergence between the estimated matrix and thetarget spectrogram, where the overview of the model is asshown in Figure 1. First, let Y be a matrix of acoustic signals
having those of musical instruments to be separated. Then,
we decompose Y into the matrix H with bases for musicalinstrument parts, and that U with temporal activation for thosebases. The framework of sound source separation by NMF isformalized as minimizing the objective function in (1) [9],C(θ)=D(θ) + Rp (θ) + Rh (θ) + Rb (θ) + Rl (θ) (1)where each formula in the right hand side is as listed below:
D(θ)
Rp (θ)Rh (θ)
Rb (θ)Rl (θ)

divergence between the estimated matrix and thetarget spectrogram.
the cost function for separating non-harmonic
musical instruments.
the cost function that is common for separating
harmonic musical instruments.
the cost function for separating backing harmonic
musical instruments.
the cost function for separating lead harmonic
musical instruments.

A. Separating Non-harmonic Instruments
For each non-harmonic musical instrument, we prepare twobases, one of which is for non-harmonic sound that is smoothin frequency direction, while the other is for harmonic soundaccompanying the non-harmonic sound. We prepare the basefor the harmonic sound because even non-harmonic musicalinstruments contain harmonic sound such as the interval of
the tom and the snare.
More specifically, we prepare bases with the constraint of
having smooth envelope in frequency direction, and other baseswith restricted activation. Let I pi (i = 1, . . . , q) (corresponding
to the H matrix PaIi and Pf Ii in Figure 1) denote nonharmonic musical instruments, and P ahi and Pf hi denoteindices of bases for those non-harmonic musical instruments.
Then, the following equations give how to initialize bases of
those non-harmonic musical instruments as well as the costfunction Rp (θ) is defined: [10]hkm
Rp (θ)= 1, (m = Pahi , Pf hi )
K

= γpa
(unP ahi − un−1P f hi )2
i

+γpf

n=2

K

i

(hkP f hi − hk−1P f hi )2k=2Here, the coefficients γ pa and γpf represent relative weights ofthe first and the second terms of the cost function. The moresmooth the activation is, the smaller the cost function above
is designed to be.
B. Separating Harmonic Instruments
The general formalization of separating harmonic instruments overall follows that by Raczyński [7]. In this paper, weclassify harmonic musical instruments into those for lead suchas with playing main melody and those for backing such aswith playing chord. The underlying idea of separating leadand backing is to separate them by changing constraints in
NMF, since the lead instrument plays the single tone while
the backing instrument plays the chord sound.
Let Ihi (i = 1, . . . , r) (corresponding to the H matrix
HIi in Figure 1) denote harmonic musical instruments, andIhi denote the set of indices of bases for those harmonic
musical instruments. Then, the following equation gives how
to initialize bases of those harmonic musical instruments.

1 (mod(k − ωlowhi , ξ) = 0, ξ ∈ Hhi
hkm =0 (otherwise)
Supposing that we are given any possible central frequency
that the harmonic musical instrument I hi can play, then, withthis initialization, we can prepare bases with comb shape whichhave all the overtones of the given central frequency. Next, thecost function R h (θ) that is common for separating harmonic
musical instruments is defined as below: 
Rh (θ) = γh(hφ(m,j)m − hφ(m−1,j)m−1) )2
i,j m∈Hhi

where the coefficient γ h represents a relative weight of thecost function, and φ(m, j) represents j-th index which satisfies
hkm = 0. This cost function is meant to be that the strengths ofthe overtones of the same level of adjacent bases are designed
to be closer in a harmonic musical instrument. This designof the cost function enables the modeling of the continuouschanges of the pitch with harmonic musical instruments suchas the guitar mentioned in section III.

Next, the following sections describe how to separatebacking and lead harmonic instruments.
1) Separating Backing Instruments: Since the backing instrument plays the chord sound, it is probable that it has
more than one central frequencies. Here, we assume that thosecentral frequencies are mostly observed within a rather narrow
range in the frequency space. By expressing this assumption
through a probability distribution, those central frequencies aremostly observed very closely around the frequency bin which
gives the expectation on the frequency axis. Considering theobservation above, we design the requirement for separatingbacking instruments as having those activations which minimize their variance. Let H b (⊂ Hhi ) be the set of indices ofbases for those backing harmonic musical instruments, then thecost function R b (θ) for separating backing harmonic musicalinstruments is defined as below:
m2 umn m∈Hb
Rb (θ) = γb
− μ2
umn
n

μ =

m∈Hb

mumn

m∈Hb



where the coefficient γ b represents a relative weight of the costfunction.
2) Separating Lead Instruments: Since the lead instrument
plays the main melody, it is probable that it has one specificcentral frequency. Considering this observation, we design therequirement for separating lead instruments as having those
activations which maximize their kurtosis. Let H l (⊂ Hhi ) be
the set of indices of bases for those lead harmonic musicalinstruments, then the cost function R l (θ) for separating leadharmonic musical instruments is defined as below: 
m4 umn − 4
m3 umnRl (θ) = −γl
m∈Hl
2

+6μ



m∈Hl
2

m umn

m∈Hl

 

umn

−1

m∈Hl

4

−3μ

where the coefficient γ l represents a relative weight of the costfunction.
V.

PARAMETERS AND THEIR VALUES IN E VALUATION

parameter
Sampling rate
STFT window
overlap
central frequency range(Lead Guitar)
central frequency range(Back Guitar)
central frequency range(Bass)
γh
γpa
γpf
γl
γb

TABLE II.

value
44,100Hz
Hamming (8,192pts)
7,680
C4(261.43Hz) ∼ C7(2,093Hz)
C3(130.81Hz) ∼ C5(523.25Hz)
C2(65.406Hz) ∼ C4(261.43Hz)
16,384
1
13,1072
1.0 × 10−6
2.0 × 10−3

E VALUATION R ESULTS WITH THE MIDI S OUND S OURCE[ D B]
t001
t002
t003
t004
t005
t006
t007
t008
t009
t010
Ave

LeadG
5.442
8.973
2.921
7.908
2.025
6.968
4.967
-2.956
0.913
-1.61
3.555

BackG
11.495
8.154-2.252
1.145-2.091
4.097-31.206-0.113-2.107-0.537-1.341Ba-0.662-0.595-7.733
1.207
4.478
0.485
1.243
7.732
1.602
2.56
1.032

Dr
8.843
6.464
-5.217
6.695
7.075
6.538
10.84
11.971
2.156
4.258
5.962

Ave
6.28
5.749
-3.07
4.239
2.872
4.522
-3.539
4.158
0.641
1.168

umn

m∈Hb

n

TABLE I.

E VALUATIONThis section shows results of two experimental evaluation:
the one with the MIDI sound source, and the other with the CDsound source. In the evaluation with the MIDI sound source,
we apply the proposed technique of sound source separation
to the mixture of the MIDI sound source of each musicalinstrument, and then evaluate how well the proposed methodof sound source separation performs by referring to the MIDI
sound source of each musical instrument.

A. Evaluation with the MIDI Sound Source
1) The Procedure: As the MIDI sound source, we use tenpieces of sound signals each of which consists of the MIDI
sound sources of the lead guitar, the backing guitar, the base,and the drums of 10 seconds. After we apply the proposed
technique of sound source separation to the mixture of the
MIDI sound sources of musical instruments, and have theresults of separated sound signals, we evaluate the results withthe following SNR:
SN R=

10 log10

V ar(Ot )V ar(St )where Ot denotes the original MIDI sound source for eachmusical instrument, while S t denotes the acoustic signal obtained by decoding the difference of the spectrogram of O t
and that obtained from the result of separated sound signals.
Here, Table I lists the values of the parameters used in theevaluation of this paper.
2) Evaluation Results: Table II lists the results of evaluation for each of the musical instruments. As can be seenfrom this result, we achieve especially high performance inseparating the lead guitar and the drums. This is partiallybecause we assign a larger number of bases to the lead guitar
than to the backing guitar and the base, which enables moredetailed representation of the sound source of the lead guitar.
As for the drums, we assign three bases considering the highhat, the snare, and the bass drum, while it is not easy to decidethat we successfully separate those three sound sources by
the proposed technique. Thus, it is further necessary for us toinvent another technique to separate those sound sources of
the non-harmonic musical instruments.

Fig. 2.

Spectrogram of the CD Sound Source of Test Data

Fig. 3.

Spectrogram of the Separated Lead Guitar

Fig. 4.

Spectrogram of the Separated Backing Guitar

B. Evaluation with the CD Sound Source

We applied the proposed method to a CD sound source. The
musical piece of this CD sound source is a part with the guitar
solo for approximately ten seconds taken from 52 seconds
in ”thunder force 666” (composed by Azuma Ruriko) [11].
”Thunder force 666” is a sound source with a musical pieceof hard melodic metal including an electric guitar and drums,
which matches the proposed technique. Figure 2 shows the
spectrogram of this CD sound source. Figures 3 ∼ 6 show the
spectrograms of the separated lead guitar, the backing guitar,
the base, and the drums, while Figure 7 shows that of theirmixed down data. It is obvious from this result that separationis performed with a precision as high as that of the evaluation
with the MIDI sound source.

VI.

C ONCLUSION AND F UTURE W ORKThis paper proposed a method to separate polyphonicmusic signal into signals of each musical instrument by NMF
based on preservation of spectrum envelope. Based on theevaluation results presented in this paper, it is definitely
required that we intensively evaluate the proposed method witha much larger number of sound sources including those listed
in the Metal Songs Top 100.
R EFERENCES[1] K. Itoyama, M. Goto, K. Komatani, T. Ogata, and H. Okuno, “Simultaneous realization of score-informed sound source separation ofpolyphonic musical signals and constrained parameter estimation for
integrated model of harmonic and inharmonic structure,” Journal of
Information Processing Society of Japan, vol. 49, no. 3, pp. 1465–1479,
2008, (in Japanese).
[2] N. Yasuraoka, T. Yoshioka, K. Itoyama, T. Takahashi, K. Komatani,
T. Ogata, and H. Okuno, “Musical sound separation and synthesis usingFig. 5.

Spectrogram of the Separated BaseFig. 6.

Spectrogram of the Separated Drums

Fig. 7.

Spectrogram of the Mixed Down Data of Figures 3 ∼ 6

harmonic/inharmonic GMM and NMF for phrase replacing system,”
Journal of Information Processing Society of Japan, vol. 52, no. 12,
pp. 3839–3852, 2011, (in Japanese).
[3]

M. Nakano, Y. Kitano, J. Le Roux, H. Kameoka, N. Ono, and
S. Sagayama, “Polyphonic music signal analysis using non-negative
matrix factorization with deformable bases,” IPSJ Special Interest
Group on Music and Computer, vol. 2010-MUS-84, no. 10, pp. 1–6,
2010, (in Japanese).

[4]

M. Goto, “A real-time music-scene-description system: PredominantF0 estimation for detecting melody and bass lines in real-world audio
signals,” Speech Communication, vol. 43, no. 4, pp. 311–329, 2004.

[5]

K. Itoyama, M. Goto, K. Komatani, T. Ogata, and H. Okuno, “Constrained parameter estimation of harmonic and inharmonic models forseparating polyphonic musical audio signals,” IPSJ Special Interest
Group on Music and Computer, vol. 2007-MUS-37, pp. 81–88, 2007,(in Japanese).

[6]

Y. Uchiyama, K. Miyamono, T. Nishimoto, N. Ono, and S. Sagayama,“Automatic chord detection using harmonic/percussive sound separationfrom music acoustic signals,” IPSJ Special Interest Group on Music andComputer, vol. 2008-MUS-76, no. 23, pp. 137–142, 2008, (in Japanese).

[7] S. A. Raczy ński, N. Ono, and S. Sagayama, “Multipitch analysis withharmonic nonnegative matrix approximation,” in Proc. ISMIR, 2007,
pp. 381–386.
[8] M. N. Schmidt and M. Mrup, “Nonnegative matrix factor 2-d deconvolution for blind single channel source separation,” in Proc. ICA, 2006,
pp. 700–707.
[9] H. Kameoka, “Non-negative matrix factorization and its applications toaudio signal processing,” Acoustical Science and Technology, vol. 68,
no. 11, pp. 559–565, 2012.
[10] T. Virtanen, “Monaural sound source separation by nonnegative matrix
factorization with temporal continuity and sparseness criteria,” IEEE
Transaction on Audio, Speech and Language Processing, vol. 15, no. 3,
pp. 1066–1074, 2007.
[11] R. Azuma, “The God of Melodicspeedmetal,” http://metaldtm.com/.