INTERSPEECH 2017
August 20–24, 2017, Stockholm, Sweden

Speech enhancement using non-negative spectrogram models with
mel-generalized cepstral regularization
Li Li1 , Hirokazu Kameoka2 , Tomoki Toda3 and Shoji Makino1
1

2

University of Tsukuba, JapanNTT Communication Science Laboratories, NTT Corporation, Japan
3
Information Technology Center, Nagoya University, Japanlili@mmlab.cs.tsukuba.ac.jp, kameoka.hirokazu@lab.ntt.co.jp
tomoki@icts.nagoya-u.ac.jp, maki@tara.tsukuba.ac.jp

Abstractand voice conversion does not always lead to satisfactory results. To overcome this drawback, we have previously proposeda semi-supervised NMF framework using cepstral distance regularization (CDRNMF) [4], which makes it possible to enhancespeech in both the spectral and cepstral domains by optimizing
a combined objective function of an NMF-based model ﬁtting
criterion deﬁned in the spectral domain and a Gaussian mixture model-based probability distribution deﬁned in the melfrequency cepstral coefﬁcients (MFCC) domain. The objectivefunction that contains the explicit form of the mapping function
from the spectral domain to the MFCC domain becomes very
difﬁcult to optimize, resulting in an optimization algorithm withslow convergence.
This paper proposes yet another formulation with an objective function that is relatively easy to optimize. Speciﬁcally, we combine an NMF-based model ﬁtting criterion witha divergence measure between the NMF model and the melgeneralized cepstral (MGC) representation [5] of a prototype
spectrum in a pretrained codebook. An MGC representation isa parametric model for a spectral envelope described by a setof coefﬁcients called the MGC coefﬁcients, which takes the allpole spectral model and the cepstral representation as special
cases. Through the hyperparameter setting, the MGC representation can be set to have a mel-scale frequency resolution similar to the human auditory system. This hyperparameter setting
makes the MGC representation an auditory-motivated representation of audio signals and is widely used in parametric speechsynthesis. This motivates us to believe that the proposed method
will also have an effect in enhancing the preceived quality of thegenerated speech.

Spectral domain speech enhancement algorithms based on nonnegative spectrogram models such as non-negative matrix factorization (NMF) and non-negative matrix factor deconvolution
are powerful in terms of signal recovery accuracy, however they
do not directly lead to an enhancement in the feature domain(e.g., cepstral domain) or in terms of perceived quality. We havepreviously proposed a method that makes it possible to enhancespeech in the spectral and cepstral domains simultaneously. Although this method was shown to be effective, the devised algorithm was computationally demanding. This paper proposes
yet another formulation that allows for a fast implementation
by replacing the regularization term with a divergence measure between the NMF model and the mel-generalized cepstral(MGC) representation of the target spectrum. Since the MGC is
an auditory-motivated representation of an audio signal widelyused in parametric speech synthesis, we also expect the proposed method to have an effect in enhancing the perceived quality. Experimental results revealed the effectiveness of the proposed method in terms of both the signal-to-distortion ratio andthe cepstral distance.
Index Terms: speech enhancement, non-negative matrix factorization, mel-generalized cepstral representation, single channel signal processing

1. Introduction
Enhancing speech signals from observed noisy speech signals is
an important task in various applications since the presence ofnoise can signiﬁcantly degrade the quality of speech transmission systems and the performance of e.g., speech recognitionand speech conversion.
In recent years, many spectral domain speech enhancement algorithms based on non-negative spectrogram models,such as non-negative matrix factorization (NMF) [1,2] and nonnegative factor deconvolution [3], have been proposed. Thesemethods are particularly noteworthy in that they can work successfully even without any prior knowledge about the noisecharacteristics by using a reasonable assumption that speechand noise are additive in the spectral domain. In a semisupervised setting, the aim of these methods is to approximate
the magnitude (or power) spectrogram of an observed mixture
signal as the sum of speech and noise spectrogram models using the pretrained spectral templates of speech. The underlying
speech components can be separated out using the Wiener ﬁlter constructed by the estimated power spectrograms of speechand noise. Although these methods are shown to be effective
in terms of signal recovery accuracy, they do not directly leadto an enhancement in the feature domain (e.g., cepstral domain)or in terms of perceived quality. Hence, naively using these algorithms as a front-end processing for e.g., speech recognition

Copyright © 2017 ISCA

2. Formulation
2.1. Speech enhancement with NMF
We start by reviewing a speech enhancement method using
NMF. Given an observed power spectrogram of a noisy speech
signal Y = (Yω,t )Ω×T ∈ R≥0,Ω×T , where ω and t are frequency and time indices, we consider approximating it by the(s)(n)
sum of speech and noise components, Xω,t = Xω,t + Xω,t ,(s)(n)
where Xω,t and Xω,t are represented by the non-negative lin(s)(s)
ear combination of Ks speech basis spectra W1,ω , . . . , WKs ,ω
(n)

(n)

and Kn noise basis spectra W1,ω , . . . , WKn ,ω :(s)Xω,t =

Ks

k=1

(s)

(s)

(n)

Wk,ω Hk,t , Xω,t =

Kn


(n)

(n)

Wk,ω Hk,t .

(1)

k=1

In a semi-supervised setting, the speech basis spectra are pretrained using clean speech training samples. W (n) , H (s) andH (n) are the variables to be estimated at test time. Here, we

1998

http://dx.doi.org/10.21437/Interspeech.2017-1492

(s)Table 1: Examples of α and γ settings under 16 kHz sampling
frequency.
γ = −1All-pole model
Warped all-pole model

α=0
α = 0.42of Xω,t are also enhanced. Here, we introduce a penalty term(s)deﬁned as the Itakura-Saito (IS) divergence [7] between Xω,tand Sω,t (θ)γ=0
Cepstral representation
Mel-cepstral representation

J (W (s) , H (s) , θ) =use the generalized Kullback Leibler (KL) divergence


Yω,t
Yω,t log− Yω,t + Xω,t ,I(Y |X) =
Xω,t
ω,t
(s)

ω,t

(n)

3.1. Update rules based on majorization-minimization(3)

Suppose F (Θ) is an objective function that we wish to minimize with respect to Θ. Majorization-minimization principle
considers to construct a “majorizer” F + (Θ, α) deﬁned as afunction satisfying F (Θ) = minα F + (Θ, α), where α is called
an auxiliary parameter. An algorithm that consists of iteratively
minimizing F + (Θ, α) with respect to Θ and α is guaranteed
to converge to a stationary point of the objective function. Itshould be noted that this concept is adopted in many existingalgorithms [1, 10].
Here, we derive a majorizer for the objective function (7)with respect to W and H. First, I(Y |X) involves a “log-ofsum” form of Wk,ω Hk,t . Since the negative logarithm function
is a convex function, we can invoke Jensen’s inequality to construct an upper bound of I(Y |X) having a “sum-of-logs” formin the same way as [1]

⎧
1/γ
M
⎪
jω
⎪
⎨ 1+γ
c(m)Ψm(0 < |γ| ≤ 1)α (e )
m=0,(γ = 0)m=0where the coefﬁcients c = [c(0), . . . , c(M )]T are called the
MGC coefﬁcients (MGCC). The function lγ−1 (· ) is the inverseof the generalized logarithmic function(ω γ − 1)/γ
log ω

(7)

In this section, we derive optimization algorithms for W (n) ,H (s) and H (n) based on the majorization-minimization (MM)principle [8, 9] when target MGC representation Sω,t (θ̂) with
θ ﬁxed to θ̂. The ﬁrst algorithm considers a frame-independent
representation of Sω,t (θ) whereas the the second algorithm considers Sω,t (θ) as a prototype spectrum drawn from a codebookof MGCCs pretrained using the K-means clustering.

m=0

lγ (ω) =(6)

3. Algorithms

jω
c(m)Ψmα (e )

M
⎪
jω
⎪
⎩exp
c(m)Ψmα (e )

Sω,t (θ)

−1 ,where λ ≥ 0 weighs the importance of the mel-generalized
cepstral regularization term relative to the NMF cost.

The mel-generalized cepstral (MGC) representation S(ω) is aparametric model for spectral envelopes of speech described by
M + 1 coefﬁcients and two hyperparameters γ and α:

=

Xω,t

F (W , H, θ) = I(Y |X) + λJ (W(s) , H(s) , θ),

2.2. Mel-generalized cepstral representation

MSω,t (θ)

(s)

− log

where Sω,t (θ) is the target MGC representation at frame t andfrequency ω deﬁned using the expression (3) described by parameter θ. In sec. 3, we introduce two ways to model Sω,t (θ),
which leads to different θ representations and algorithms.
The proposed method considers an optimization problem ofminimizing a combined objective function of (2) and (6)(2)

as a goodness-of-ﬁt criterion. Once Xω,t and Xω,t are estimated, the enhanced speech can be separated out using the
Wiener ﬁlter constructed with the estimated power spectrogramof speech and noise.
As stated above, this method does not directly lead to an enhancement in the cepstral domain, implying that naively using
it as a front-end for e.g., speech recognition and voice conversion does not always lead to satisfactory results. To address
this drawback, we introduce a novel regularization method using the MGC representation of a prototype spectrum in a pretrained codebook with the aim of improving the quality of theenhanced speech in the cepstral domain.

S(ω) = lγ−1(s)
  Xω,t(0 < |γ| ≤ 1),
(γ = 0)(4)

parameterized by γ. Ψα (z) is an all-pass function given byz −1 − αΨα (z) =,1 − αz −1(8)I(Y |X) ≤ I + (Y |X)
Wk,ω Hk,tc
I + (Y |X) =ζk,ω,t log+ Xω,t ,− Yω,tζk,ω,tω,t(5)kwhich can be seen as a frequency warping function parameterized by α. Here, α must satisfy |α| < 1. We can conﬁrm that S(ω) takes the all-pole spectral model and the cepstral representation as special cases when (γ, α) = (−1, 0) and(γ, α) = (0, 0), respectively. When the sampling frequency
is 16 kHz, the phase characteristic of the all-pass function becomes a good approximation to the mel scale with α = 0.42
and to the bark scale with α = 0.55 [6]. Tab. 1 shows the four(γ, α) settings we investigated in our experiments.

where =c denotes equality up to a constant term and ζk,ω,t isa positive weight that sums to unity, k ζk,ω,t = 1. It can beshown that equality of (8) holds if and only if
ζk,ω,t =Wk,ω Hk,t.
k Wk ,ω Hk ,t(9)

Next, we focus on the regularization term (6). We can see thatthe second term in (6) also has a “log-of-sum” form and thus we
can employ Jensen’s inequality

2.3. Objective function(s)

When estimating the speech spectrogram Xω,t in a mixture
spectrogram Yω,t , we would want to ensure that the MGCCs

J (W (s) , H (s) ; θ̂) ≤ J + (W (s) , H (s) , η; θ̂)

1999

(10)

Algorithm 1 Algorithm presented in subsec. 3.2

Algorithm 2 Algorithm presented in subsec. 3.3(s)Require: pretrained speech basis W (s) and I MGC prototypes
μ, parameters λ and M axIter
1: random initialize W (n) , H (s) and H (n)2: for iter = 1 to M axIter do
3:
if iter ≤ 50 then4:
update W (n) , H (s) and H (n) using SSNMF
5:
else
6:
X (s) = W (s) H (s)7:for Frame t = 1 to T do
8:compute β̂t,i using (19)9:r̂t = arg minrt J (W (s) , H (s) , θ)
10:Sω,t (θ̂) = β̂t,r̂t μω,r̂t
11:
end for12:
update H (s) and H (n) using (15) and (16)13:
update W (n) using (14)14:
end if
15: end for

Require: pretrained speech basis W , parameters λ, α, γand M , M axIter
1: randomly initialize W (n) , H (s) and H (n)2: for iter = 1 to M axIter do
3:
if iter ≤ 50 then4:
update W (n) , H (s) and H (n) using SSNMF
5:
else
6:
X (s) = W (s) H (s)7:c = M GC(X (s) , M, α, γ)
8:compute Sω,t (θ) using (17)9:update H (s) and H (n) using (15) and (16)10:
update W (n) using (14)11:
end if
12: end for
J + (W (s) , H (s) , η; θ̂)
c

=

(s)



Xω,t

ω,t

Sω,t (θ̂)

−

Ks


(s)

ηk.ω,t logk(s)Wk,ω Hk,t,ηk,ω,tHere, the parameter θ = c = {cm,t } is a set of MGCCs, whichmust be estimated in a data-driven manner. This regularization(s)aims to ensure the smoothness of the spectral envelopes of Xω,t .

where ηk,ω,t > 0 is a weight parameter that sums to unity,k ηk,ω,t = 1. It can be shown that equality of this inequality
holds if and only if(s)ηk,ω,t =3.3. Regularized algorithm using I MGC prototypes(s)Wk,ω Hk,t

.
(s)

(s)
k Wk ,ω Hk ,t(11)Here, we introduce another algorithm, which utilizes I MGC
prototypes of clean speech pretrained using the K-means algorithm. We use the following objective function for the prototype
training

The upper bound for the objective function can be easily obtained by combining the majorizers for each term as
F + (W , H; θ̂) = I + (Y |X) + λJ + (W (s) , H (s) , η; θ̂). (12)CIS (μ, r) =The update rules for Wω,k and Hk,t can be obtained by
setting at zeros the partial derivatives of the derived majorizer
(s)
(n)
(s)
(n)
with respect to Wk,ω , Wk,ω , Hk,t and Hk,t :
(s)

(s)

Wω,t ← Wω,t

ω,t

(13)
(n)

(n)

Wω,t ← Wω,t
(s)

(s)

Hω,t ← Hk,t

(14)

(s)
(s)
(s)
ω Yω,t Wk,ω /Xω,t + λω Wk,ω /Xω,t
,
(s)
(s)
ω Wk,ω + λω Wk,ω /Sω,t (θ̂)

(15)
(n)

(n)

Hω,t ← Hk,t(n)
ω Yω,t Wk,ω /Xω,t.
(n)ω Wk,ω

(s)

(s)

Yω,t
Yω,t
− log
−1
μω,rt
μω,rt

,

(18)

where μω,i denotes the prototype spectrum associated with(s)cluster i, Yω,t denotes the power spectrum of a clean speechsample and rt ∈ {1, . . . , I} denotes a cluster indicator variable,
describing to which of the I clusters the t-th speech spectrum
is assigned. Note that this objective differs from the regular Kmeans algorithm in that the distance between a data sample anda cluster prototype is measured using the IS divergence.
At test time, we ﬁnd the prototype spectrum μi closest(s)to X t in terms of the IS divergence at each iteration whereSω,t (θ) = βt,rt μω,rt . Here, the parameter θ = {rt , βt,rt }consists of a set of cluster indicator variables and corresponding
scaling parameters introduced to eliminate the scaling indeter(s)minacy. When Xω,t and μω,i are given, we can easily obtain
the optimal scaling as

(s)
(s)
(s)
t Yω,t Hk,t /Xω,t + λt Hk,t /Xω,t
,
(s)
(s)
t Hk,t + λt Hk,t /Sω,t (θ̂)(n)
t Yω,t Hk,t /Xω,t,(n)t Hk,t



(16)

(s)

β̂t,i =

Since the regularization term only impacts on the estimation of(n)(n)
speech components, the update rules for Wk,ω and Hk,t are thesame as the regular NMF algorithm [1].

1  Xω,t.
Ω ω μω,i(19)

4. Experiments

3.2. Regularized algorithm using frame-independent MGC
representation

To evaluate the effect of the proposed method for speech enhancement task, we tested the baseline semi-supervised NMF(SSNMF) [2], and the proposed algorithm 1 and 2 using the
speech data excerpted from the ATR503 database [11] andtwo types of measured noise, namely departement store andsubway station, excerpted from the ATR ambient noise sounddatabase. We used Signal-to-distortion ratios (SDRs), signalto-interference ratios (SIRs) [12] and MFCC distance for the

In the rest of sec. 3, we introduce two approaches to modelingthe target MGC representation S.
Algorithm 1 shows an algorithm using the frameindependent MGC regularization. Here, Sω,t (θ) is deﬁned as


M
jω
cm,t Ψm(17)
Sω,t (θ) = lγ−1α (e ) .
m=02000

Table 2: SDR, SIR and MFCC distance improvement [dB] obtained by the baseline SSNMF and the proposed algorithms under noise
measured in a department store and a subway station. The highest score of each term is shown in bold type.

SSNMF (baseline)(-1,0.42)(-1,0)(0,0.42)(0,0)(-1,0.42)(-1,0)(0,0.42)(0,0)(-1,0.42)(-1,0)(0,0.42)(0,0)

Order = 12
Algorithm 1
Order = 20

Algorithm 2

Objective function

7

6.6
6.4
6.2
6

7.5

200

400

600

800

1000

Iteration

1200

1400

1600

1800

2000

1400

1600

1800

2000

Convergence Behavior of Algorithm 2

×104

7
6.5
6
5.5
0

200

400

600

800

1000

Iteration

1200

Figure 1: Convergence behavior of the proposed algorithms.
Algorithm 1 is shown in the top ﬁgure and algorithm 2 is shownin the bottom one.
Table 3: Computational time of SSNMF, CDRNMF and the proposed methods.
SSNMF [2]CDRNMF [4]Algorithm 1
Algorithm 2

rumtime/iter [sec]0.02970.43170.81813.4218Input SNR = 0 dB
Δ SDR Δ SIR Δ MFCC
3.64
5.32
1.30
3.38
6.95
1.44
3.28
8.24
1.34
3.57
5.39
1.21
3.28
5.71
1.14
3.40
6.41
1.54
3.24
7.69
1.65
4.05
5.58
1.34
3.75
5.66
1.20
4.25
8.02
1.86
3.86
7.11
1.80
4.09
7.30
1.78
4.12
6.77
1.62

iterations SSNMF at the beginning of every algorithm as theinitialization.
First, we show the convergence behavior of the proposed
algorithms. For comparsion, we also show the runtime of every
iteration and iteration demanding of SSNMF and CDRNMF in
Tab. 3. All the programs were run in the MATLAB 2015b with
Inter(R) Core(TM) i5-5200U CPU @2.20GHz 64bit and 8.0
GB memory. Fig. 1 shows the proposed algorithms converged
quickly. Comparing with CDRNMF, though the time of every
iteration became longer, the processing time decreased signiﬁcantly since the proposed algorithms converged with fewer iterations.
Tab. 2 shows the results of SDR, SIR and MFCC distance
improvement [dB] obtained by SSNMF and the proposed algorithms under two types of noise. Both of the proposed algorithms achieved an improvement in all the evaluation criteria with appropriate hyperparameters (λ, γ, α), which showed
the effectiveness of the proposed algorithms in jointly enhancing speech in spectral and cepstral domain. As shown in the
Tab. 2, the algorithm 2 achieved higher SDR, SIR and MFCC
distance improvement on average than algorithm 1. Although(γ, α) = (−1, 0.42) did not achieve the highest score under all
the conditions, it still can be thought as the ﬁrst choice of hyperparameters since the stationary improvement shown in thetable.

Convergence Behavior of Algorithm 1

×105

6.8

5.8
0

Objective function

Order = 20

Input SNR = -5 dB
Δ SDR Δ SIR Δ MFCC
4.05
5.23
1.63
5.21
7.37
1.87
5.10
7.26
2.06
4.64
6.29
1.70
4.96
6.70
1.83
4.69
7.22
1.99
4.98
6.87
1.97
3.83
5.11
1.32
4.70
6.66
1.61
5.00
8.01
2.54
4.41
8.32
2.41
4.70
7.83
2.27
4.59
8.02
1.86

iteratin number
1000
5000002000
2000

5. Conclusions

evaluation. Given two D-dimension MFCC sequences x[d] andy[d] calculated from N frequency bins, the MFCC distance isdeﬁned as follow:

 D

20D 
2
Dist =(x[d] − y[d])2 .
(20)
N ln 10

This paper proposed a novel approach to jointly enhance speechin spectral and cepstral domain, which allows for a fast implementation. The method optimizes a combined objective function of an NMF-based model ﬁtting criterion with a divergencemeasure between the NMF model and the mel-generalized cepstral representation in the spectral domain. We derived two algorithms based on majorization-minimization principle. The
experimental results showed that both of the proposed algorithms outperformed the baseline SSNMF in SDR, SIR and
MFCC distance improvement, which showed the effectivenessof mel-generalized cepstral regularization.

d

The test data were created by adding noise signals to clean
speech signals with the signal-to-noise ratios (SNRs) of -5,
0 dB. All the audio signals were monaural and sampled at
16KHz. The STFT was computed using the Hanning window with 32ms long and 16ms overlap. In the training phase,
200 utterances spoken by 2 male and 2 female speakers wereused to train 50 speech basis spectra. For noise we used thesame number of basis spectra. The same train set was also
used for K-means training. The cluster number was set at1000. We investigated MGCC with 12 order and 20 order
repectively with four special cases shown as hyperparameters(γ, α) = {(−1, 0.42), (−1, 0), (0, 0.42), (0, 0)}. We run 50

6. Acknowledgements
This work was supported by JSPS KAKENHI Grant Number
26730100 and 17H01763, and SECOM Science and Technology Foundation.

2001

7. References[1] D. D. Lee and H. S. Seung, “Algorithms for nonnegative matrix
factorization,” in Adv. NIPS, pp. 556–562, 2000.
[2] P. Smaragdis, B. Raj and M. Shashanka, “Supervised and semisupervised separation of sounds from single-channel mixtures,” inProc. ICA 2007, pp. 414–421, 2007.
[3] P. Smaragdis, “Non-negative matrix factor deconvolution; extraction of multiple sound sources from monophonic inputs,” in International Conference on Independent Component Analysis andSignal Separation, pp. 494–499, Springer Berlin Heidelberg, 2004.
[4] L. Li, H. Kameoka, T. Higuchi and H. Saruwatari, “SemiSupervised Joint Enhancement of Spectral and Cepstral Sequencesof Noisy Speech,” in Proc. Interspeech 2016, pp. 3753–3757, 2016.
[5] K. Tokuda, T. Kobayashi, T. Masuko and S. Imai, “Mel-generalized
cepstral analysis-a uniﬁed approach to speech spectral estimation,”in ICSLP, Vol. 94, pp. 18–22, 1994.
[6] T. Masuko, “HMM-based speech synthesis and its applications,”
Institute of Technology, 2002.
[7] F. Itakura and S. Saito, “Analysis synthesis telephony based on
the maximum likelihood method,” in Proc. the 6th International
Congress on Acoustics, pp. 17–20, 1968.
[8] J.D. Leeuw, and W.J. Heiser, “Convergence of correction matrix
algorithms for multidimensional scaling,” in Geometric representations of relational data, pp. 735–752, 1977.
[9] D.R. Hunter, and K. Lange, “A tutorial on MM algorithms,” The
American Statistician, 58 (1), pp. 30–37, 2004.
[10] M. Nakano, H. Kameoka, J. Le Roux, Y. Kitano, N. Ono, and S.
Sagayama, “Convergence-guaranteed multiplicative algorithms for
non-negative matrix factorization with beta-divergence,” in Proc.
MLSP, pp. 283–288, 2010.
[11] A. Kurematsu, K. Takeda, Y. Sagisaka, S. Katagiri, H. Kuwabara,
and K. Shikano, “ATR Japanese speech database as a tool ofspeech recognition and synthesis,” Speech Communication, vol. 9,
pp. 357–363, 1990.
[12] E. Vincent, R. Gribonval, and C. Fevotte, “Performance measurement in blind audio source separation.”, IEEE transactions on audio, speech, and language processing, vol. 14, no. 4, pp. 1462–
1469, 2016.

2002