DISCRIMINATIVE NON-NEGATIVE MATRIX FACTORIZATION WITH
MAJORIZATION-MINIMIZATION
Li Li1 , Hirokazu Kameoka2 and Shoji Makino1
1

University of Tsukuba, JapanNTT Communication Science Laboratories, NTT Corporation, Japanlili@mmlab.cs.tsukuba.ac.jp, kameoka.hirokazu@lab.ntt.co.jp, maki@tara.tsukuba.ac.jp
2

ABSTRACT
Non-negative matrix factorization (NMF) is a powerful approach to single channel audio source separation. In a supervised setting, NMF is first applied to train the basis spectra ofeach sound source. At test time, NMF is applied to the spectrogram of a mixture signal using the pretrained spectra. The
source signals can then be separated out using a Wiener filter.
A typical way to train the basis spectra of each source is tominimize the objective function of NMF. However, the basis
spectra obtained in this way do not ensure that the separated
signal will be optimal at test time due to the inconsistency
between the objective functions for training and separation(Wiener filtering). To address this, a framework called discriminative NMF (DNMF) has recently been proposed. In
in this work a multiplicative update algorithm was proposed
for the basis training, however one drawback is that the convergence is not guaranteed. To overcome this drawback, thispaper proposes using a majorization-minimization principle
to develop a convergence-guaranteed algorithm for DNMF.
Experimental results showed that the proposed algorithm outperformed standard NMF and DNMF using a multiplicative
update algorithm as regards both the signal-to-distortion andsignal-to-interference ratios.
Index Terms— Discriminative non-negative matrix factorization, majorization-minimization, single channel, speechenhancement
1. INTRODUCTION
Single channel audio source separation is the challenging
task of extracting individual source signals from a monaural
recording of a mixture signal. Non-negative matrix factorization (NMF) [1, 2] has attracted a lot of attention in recent
years after being proposed as a powerful approach for audio
source separation.
Factorizing an observed magnitude (or power) spectrogram of a mixture signal, interpreted as a non-negative matrix, into the product of two non-negative matrices amounts
to approximating the observed spectra by a linear sumof basis spectra scaled by time-varying amplitudes. In a
supervised/semi-supervised setting, NMF is first used to train
the basis spectra of each sound source using individually
recorded audio samples. At test time, NMF is applied tothe spectrogram of a test mixture signal, where each subset
This work was supported by JSPS KAKENHI under Grants 26730100
and 26280060, and SECOM Science and Technology Foundation.

978-1-5090-5925-6/17/$31.00 ©2017 IEEE141

of the basis spectra is fixed at the pretrained spectra. The
source signals can then be separated out using a Wiener filter
constructed by employing the estimated power spectrogramof each source. A typical way to train the basis spectra ofeach source is to minimize a divergence measure between the
NMF model and the spectrogram of the training samples ofthat source. However, the basis spectra obtained in this way
do not ensure that the separated signal at test time will be
optimal since the objective functions for training and separation are inconsistent. To address this, a framework called1
discriminative NMF (DNMF) has recently been proposed[3]. The central idea of DNMF is that the basis spectra aretrained in such a way that the output of the Wiener filter
becomes as close to the spectrogram of each of the training
examples as possible so that the separated signals become
optimal at test time. This approach differs from the conventional supervised NMF framework in that it uses the training
examples of all the sources to train the basis spectra for eachof the sources. This is important since it helps to enhancethe discriminative power of the basis spectra. However, as
shown later, the training criterion for DNMF becomes analytically more complex than the typical divergence measuresused in the standard NMF framework, which causes difficulty as regards optimization of the basis spectra. In [3]Weninger proposed a multiplicative update algorithm for thebasis training, however one drawback is that the convergence
is not guaranteed. To overcome this drawback, this paper proposes using a majorization-minimization principle to derive aconvergence-guaranteed algorithm for DNMF.
2. DISCRIMINATIVE NON-NEGATIVE MATRIXFACTORIZATION
2.1. Standard NMF
We start by reviewing standard NMF for single channel
source separation. Let us denote the number of sources by L,
and an observed power spectrogram by Y = (Yω,t )Ω×T ∈R≥0,Ω×T , where ω and t are frequency and time indices.
Supervised NMF factorizes an observed spectrogram Y ofa mixture signal into the product of a non-negative basis
matrix W = [W 1 , W 2 , . . . .W L ] and a non-negative coefficient (activation) matrix H = [H 1 ; H 2 ; . . . ; H l ], where
1 While many methods called “discriminative NMF” have been proposedwith the aim of enhancing the discriminative power of the basis spectra [3]–[10], here we use this term in relation to the work done by Weninger [3].

HSCMA 2017l
W l = (Wω,k)Ω×K l ∈ R≥0,Ω×K with l = 1, 2, . . . , L
is pretrained using the spectrograms of training samplesl
S l = (Sω,t)Ω×T . A typical criterion for this isl

W l = argmin D(S l |W l H l ),

(1)

Wl

where D is a cost function that measures the difference between S l and W l H l . At test time, W is fixed at the pretrained basis spectra and the activation matrix H is estimatedso that the objective funtion
H = argmin D(Y |W H),(2)

where the concatenated basis matrix W = [W s , W n ] consists of a total of K basis vectors with Ks for speech and Kn
for noise. It should be noted that in [3] a sparse regularization
term [9] is used to promote the sparsity of H.
An inspection of (1) and (6) shows that the training criterion for DNMF is more analytically complex than the objective function of standard NMF. In [3], Weninger proposeda multiplicative update algorithm for solving the above optimization problem. The algorithm consists of two stages:
First, the activation matrix H is obtained by solving (2) using
NMF. The basis matrix W is then iteratively updated according to the following rulesH

is minimized subject to non-negativity. When D is a generalized Kullback-Leibler (KL) divergence, the objective function
can be written as
DKL (Y |W H)∑(=
Yω,t log
ω,t

)
Yω,t
− Yω,t + [W H]ω,t ,[W H]ω,t

W lH l⊗Y,WHs(W H)2Wn ← Wn ⊗(3)

where [·]i,j denotes the {i, j}-th element of a matrix. Once
W and H are obtained, the signals can be separated by a
Wiener filter constructed using the estimated power spectrogram as follows
Ŝ l =

S s ⊗W s H s
H⊗W H
Ws ← Ws ⊗ W
M ⊗W n H n(4)

where ⊗ and ·· denote elementwise multiplication and division.
2.2. DNMF and multiplicative update algorithm
Instead of using (1), Weninger [3] proposed directly using thereconstruction error of the separated signals as an objectivefunction for the basis training∑J =αl DKL (S l |Ŝ l ),(5)lwhere αl ≥ 0 weighs the importance of source signal l. This
framework is called discriminative NMF by analogy with thediscriminative models for classification or regression.
For convenience of explanation, here we consider aspeech enhancement problem where the sources are speechsand noise, S s = (Sω,t)Ω×T ∈ R≥0,Ω×T denotes trainingnsamples of clean speech and S n = (Sω,t)Ω×T ∈ R≥0,Ω×T
denotes training samples of noise. Since we are concernedwith reconstructing clean speech as well as possible, we
set α at 1 for speech and 0 for noise. In the same way
as in 2.1, we use a Wiener filter to separate the spectrogram of speech Ŝ s from the spectrogram of a mixture signal
M = (Mω,t )Ω×T ∈ R≥0,Ω×T . The discriminative training
problem can be cast as an optimization problem
(
)
s
s
s W H
minimize f (W , H) = DKL S⊗ M , (6)WH∑
Wω,k = 1,
subject to ∀k,
ω

142

s

H sT

H sT

,

nT
M ⊗W s H s(W H)2 H
.
nT
Ss
WH H

Here, the multiplicative factors are given by dividing the negative parts by the positive parts of the partial derivative off with respect to W s and W n in the same way as in [11].
Although this algorithm is easy to implement and works reasonably well in practice, one drawback is that convergence toa stationary point of f is not guaranteed.
3. DNMF WITH MAJORIZATION-MINIMIZATION
3.1. Majorization-minimization principle
To address the problem of the multiplicative update algorithm
shown above, we derive a novel convergence-guaranteed algorithm for DNMF based on a majorization-minimization(MM) principle. When constructing an MM algorithm tominimize a certain objective function, the main issue is how
to design an auxiliary function called a “majorizer” that is
guaranteed to never be below the objective function. Suppose
F (Θ) is an objective function that we wish to minimize withrespect to Θ. A majorizer F + (Θ, α) is then defined as a function satisfying F (Θ) = minα F + (Θ, α), where α is called an
auxiliary parameter. An algorithm that consists of iteratively
minimizing F + (Θ, α) with respect to Θ and α is guaranteed
to converge to a stationary point of the objective function. Itshould be noted that this concept is adopted in many existing algorithms. For example, the expectation-maximization(EM) algorithm [13] builds a surrogate for a likelihood function of latent variable models by using Jensen’s inequality.
It is also well known for its use in devising an algorithm for
NMF [1, 12]. In general, if we can build a tight majorizer thatis easy to optimize for the objective function of some optimization problem, we can expect to obtain a fast-converging
algorithm.
3.2. Majorizer for objective function
In this section, we derive a novel majorizer for the objectivefunction (6). First, let us focus on the term
∑K s

s
s
k=1 Wω,k Hk,t∑K

k=1 Wω,k Hk,t.

(7)

To construct a majorizer for this term, we can use the following inequality:
Lemma 1. For a ∈ R>0 and b ∈ R>0 ,where ηω,t is an arbitrary positive number. The equality ofthis inequality holds if and only if ηω,t = Gω,t . Since a
quadratic function is convex, we can apply Jensen’s inequality
to the third term, which yields

a
λa2
1
≤
+
.
b
2
2λb2

K
s 2 s 2
∑
Wω,k
Hk,t
s

Gsω,t 2 ≤The equality holds if and only if λ = 1/(ab).
Proof of Lemma 1. For a, b, λ ∈ R>0 ,
(
)2
(
)
1
a
1
2
λ a−
=λ a −2 + 2 2 ≥0λbλb λ b
2
a
λa
1
⇒ ≤
+
.
b
2
2λb2

k=1

βk,ω,t

,

(14)

where
∑βk,ω,t > 0 is also a positive number that sums to unity,i.e., k βk,ω,t = 1. The equality of (14) holds if and only if
s
s
Wω,k
Hk,t
.
βk,ω,t = ∑K sssk′ =1 Wω,k′ Hk′ ,t

(8)

(15)

The equality holds if and only if a − 1/(λb) = 0.
Since Mω,t is non-negative, we can construct an uppersHsbound for WW H
⊗ M according to the above lemma,()
W sH sDKL S s
⊗M
WH
)
∑(
Gsω,t
c
s
s
s
=
− Sω,t log Gω,t + Sω,t log Gω,t +
Mω,t
Gω,t
ω,t
∑(
s
s
log Gω,t≤log Gsω,t + Sω,t
− Sω,tω,t+As regards the fourth term, we can use the fact that 1/x2 isconvex in the first quadrant and use Jensen’s inequality to obtain a majorizer:
3∑ θk,ω,t
1
≤
2
2
2 ,
Gω,t
Wω,k Hk,t∑where θk,ω,t > 0 and k θk,ω,t = 1. We can confirm that theequality of this inequality holds if and only if)
λω,t Mω,t Gsω,t 2
Mω,t
+
, (9)
2
2λω,t G2ω,twhere =c denotes equality up to a constant term, Gsω,t =
∑K
∑K s
s
s
k=1 Wω,k Hk,t . The equalk=1 Wω,k Hk,t and Gω,t =ity of (9) holds if and only if
λω,t =

1.
Gsω,t Gω,t(10)

In the following, we construct a majorizer for each of theterms in the right-hand side of (9).
ssSince Sω,tis positive, −Sω,tlog Gsω,t is convex in Gsω,t .
Hence, we can use Jensen’s inequality to obtain a majorizer
for this term as− log Gsω,t ≤ −

Ks
∑
k=1

γk,ω,t log

s
s
Wω,k
Hk,t
,
γk,ω,t(11)

where γk,ω,t is a positive weight that sums to unity. The
equality of (11) holds if and only if
γk,ω,t = ∑

s
s
Wω,k
Hk,t
s
s .
k′ Wω,k′ Hk′ ,t

(16)

k

θk,ω,t = ∑Wω,k Hk,t.
k′ Wω,k′ Hk,t′(17)

From (9), (11), (14) and (16), we can construct a majorizer
for the objective function asf (W , H) ≤ f + (W , H, Γ)sss∑∑ Sω,tWω,kHk,tWω,k Hk,tsSω,tγk,ω,t log=−+
γk,ω,t
ηω,t
ω,t,k

+

∑ λω,t Mω,tω,t,k2βk,ω,t

ω,t,ks 2 s 2Wω,kHk,t +∑
ω,t,k3
Mω,t θk,ω,t2
2 + d,
2λω,t Wω,k Hk,twhere Γ denotes a set of all the auxiliary variables, λω,t ,γk,ω,t , ηω,t , βk,ω,t and θk,ω,t , and d denotes a constant term.
This majorizer is particularly noteworthy in that it can be minimized analytically with respect to Wω,k and Hk,t since it isgiven as the sum of the reciprocal, logarithmic, first-order andsecond-order functions.

(12)s
The second term Sω,tlog Gω,t is concave in Gω,t . Hence, we
can use the fact that a tangent line to the graph of a differentiable concave function lies entirely above the graph:∑ Wω,k Hk,t+ log ηω,t − 1,
(13)
log Gω,t ≤
ηω,t
k

143

3.3. Update rules
We can obtain the update rules for Wω,k and Hk,t by setting
the partial derivatives of the proposed majorizer with respect
to W s , H s , W n and H n at zero. Thus, the update rules canbe obtained as the positive solution of the following quartic

8

Table 1. SDR improvement [dB] evaluated under department
store noise (top row) and subway station (bottom row) conditions.
Input SNR
Method-6 dB -3 dB 0 dB 3 dB Avg
SNMF
5.58
5.53
5.18 4.64 5.23
DNMF MU
5.88
5.68
5.11 4.70 5.34
DNMF MM 6.41
6.29
5.72 4.70 5.78
SNMF
5.79
5.65
5.19 4.06 5.17
DNMF MU
5.51
5.86
5.22 4.80 5.35
DNMF MM 6.82
7.20
6.50 4.89 6.35

7

SDR Improvement [dB]

6
5
4
3
2
1
0

0

10

25

50

100

200

Iteration

Fig. 1.

Means and standard deviations of the SDR improvements [dB]obtained with the proposed algorithm with various iteration numbers (0, 10,25, 50, 100, 200) initialized at 5 random values.

and cubic equations:∑ λω,t Mω,t2βk,ω,t

t

−

∑

s 2
s 4
Hk,t
Wω,k
+

t

s
s 2
Sω,t
γk,ω,t Wω,k
−

t
n

ηω,t

∑ λω,t Mω,t2βk,ω,t

ω

−

∑

n 3
Wω,k
−

s 2 s 4Wω,kHk,t +ηω,ts 22λω,t Wω,k3
∑ Mω,t θk,ω,tω

n 22λω,t Wω,k

(18)
(19)

s 3
Hk,t

3
∑ Mω,t θk,ω,t
ω

n 3
Hk,t
−

ηω,t

= 0,

= 0,

s
s
∑ Sω,t
Wω,k
ω

ω

ω

n 2
2λω,t Hk,t

t

s 3
Wω,k

s 2
2λω,t Hk,t3
∑ Mω,t θk,ω,t

s
s 2
Sω,t
γk,ω,t Hk,t−ns
∑ Sω,t
Wω,k

ηω,t

3
∑ Mω,t θk,ω,tts∑ Sω,tHkn ,ttss∑ Sω,tHk,t= 0.

Table 2. SIR improvement [dB] evaluated under department
store noise (top row) and subway station conditions (bottom
row).
Input SNR
Method-6 dB -3 dB 0 dB
3 dB
Avg
SNMF
7.23
7.44
7.44
7.31
7.36
DNMF MU
8.07
7.87
7.44
7.34
7.68
DNMF MM 9.76
9.66 10.16 9.74
9.83
SNMF
7.78
8.04
8.10
8.16
8.02
DNMF MU
8.04
8.67
7.95
8.29
8.24
DNMF MM 10.77 11.58 11.89 11.28 11.38

= 0,

(20)
(21)

Although there are two quartic equations that must be solved,
it is worth noting that the parameters can be updated in parallel using these update rules. This means that this algorithm
is well suited to parallel implementations. Furthermore, sinceeach of the update rules consists of a negative 0th-order term
and a negative 2nd-order term, it turns out that there is only
one positive solution, implying that there is no need to solve
a solution selection problem.
4. EXPERIMENTS
To evaluate the effect of the proposed algorithm for speechenhancement tasks, we tested baseline supervised NMF(SNMF), DNMF using the multiplicative update algorithm
proposed in [3] (DNMF-MU) and DNMF using the proposed
algorithm (DNMF-MM) using speech data excerpted fromthe ATR503 database [14] and two types of measured noise,
namely department store and subway station noise, excerpted
from the ATR ambient noise sound database. We used signalto-distortion ratios (SDRs) and signal-to-interference ratios(SIRs) [15] for the evaluation.
The test data were created by adding noise signals to clean
speech signals with signal-to-noise ratios (SNRs) of -6, -3, 0,
and 3 dB. All the audio signals were monaural and sampled at

144

16KHz. The STFT was computed using a Hanning windowthat was 32ms long with a 16ms overlap.
In the training phase, 200 utterances spoken by 2 male and2 female speakers were used to train 40 speech basis spectra.
For noise we used the same number of basis spectra. Fig.
1 shows the means and standard deviations of the SDR improvements [dB] obtained with the proposed algorithm withvarious iteration numbers (0, 10, 25, 50, 100, 200) initialized
at 5 random values. On the basis of these results, we set theiteration number at 25 in the following experiments. We used
40 utterances selected randomly from the ATR database as thetest set.
Tabs. 1 and 2 show the results of the SDR and SIR improvements obtained with the proposed algorithm (DNMFMM) and the other two algorithms (SNMF, DNMF-MU) using two types of noise. The proposed algorithm (DNMF-MM)
yielded an SDR improvement that was 0.86 dB higher than
SNMF and 0.71 dB higher than DNMF-MU. It is worth noting that the proposed algorithm obtained an average SIR improvement of more than a 2.9 dB over SNMF, showing thediscriminative power of the basis spectra.
5. CONCLUSION
While DNMF is noteworthy in that it directly uses the reconstruction errors of the separated signals as the training criteria, it causes difficulty as regards optimization. This paper
derived a novel majorizer for the objective function of DNMF
and successfully developed an MM algorithm that is guaranteed to converge to a stationary point. Experimental results
showed that the proposed algorithm achieved significant improvements in terms of the SDR and SIR criteria over standard NMF and DNMF using the multiplicative update algorithm.

6. REFERENCES[1] D. D. Lee, and H. S. Sung, “Algorithms for non-negative
matrix factorization.”, In Advances in neural information
processing systems, pp. 556–562, 2001.
[2] P. Smaragdis, R. Bhiksha, and S. Madhusudana, “Supervised and semi-supervised separation of sounds fromsingle-channel mixtures.”, In Proc. ICA, pp. 414–421,
2007.
[3] F. Weninger, J. L. Roux, J. R. Hershey, and S. Watanabe, “Discriminative NMF and its application to singlechannel source separation.”, In Proc. INTERSPEECH,
pp. 865–869, 2014.
[4] E. M. Grais, and H. Erdogan, “Discriminative nonnegative dictionary learning using cross-coherence penalties
for single channel source separation.”, in Proc. INTERSPEECH, pp. 808–812, 2013.
[5] G. Bao, Y. Xu, and Z. Ye, “Learning a discriminative dictionary for single-channel speech separation.”,IEEE/ACM Transactions on Audio, Speech, and Language Processing 22(7), pp. 1130–1138, 2014.
[6] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, “Discriminative non-negative matrix factorization
for multiple pitch estimation.” In Proc. ISMIR, pp. 205–
210, 2012.
[7] Z. Wang, and F. Sha, “Discriminative non-negative matrix factorization for single-channel speech separation.”
In Proc. ICASSP, pp. 3749–3753, 2014.
[8] K. Kwon, J. W. Shin, and N. S. Kim, “Target source separation based on discriminative nonnegative matrix factorization incorporating cross-reconstruction error.”, IEICE Transactions on Information and Systems 98(11),pp. 2017–2020, 2015.
[9] J. Eggert, and E. Korner, “Sparse coding and NMF.”, in
Proc. of Neural Networks, vol 4, pp. 2529–2533, 2004.
[10] P. Sprechmann, A. M. Bronstein, and G. Sapiro, “Supervised non-Euclidean sparse NMF via bilevel optimization with applications to speech enhancement.”, In Proc.
HSCMA, pp. 11–15, 2014.
[11] C. Fevotte, N. Bertin, and J. L. Durrieu, “Nonnegative
matrix factorization with the Itakura-Saito divergence:With application to music analysis.”, Neural computation,
vol 21, no. 3, pp. 793–830, 2009.
[12] M. Nakano, H. Kameoka, J. Le Roux, Y. Kitano, N.
Ono, and S. Sagayama, “Convergence-guaranteed multiplicative algorithms for non-negative matrix factorization with beta-divergence,” in Proc. MLSP, pp. 283–288,
2010.
[13] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” Journal of Royal Statistical Society Series B, vol.
39, pp. 1–38, 1977.

145[14] A. Kurematsu, K. Takeda, Y. Sagisaka, S. Katagiri,
H. Kuwabara, and K. Shikano, “ATR Japanese speech
database as a tool of speech recognition and synthesis,”
Speech Communication, vol. 9, pp. 357–363, 1990.
[15] E. Vincent, R. Gribonval, and C. Fevotte, “Performancemeasurement in blind audio source separation.”, IEEE
transactions on audio, speech, and language processing,vol. 14, no. 4, pp. 1462–1469, 2016.