2017 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING, SEPT. 25–28, 2017, TOKYO, JAPANMISSING COMPONENT RESTORATION FOR MASKED SPEECH SIGNALSBASED ON TIME-DOMAIN SPECTROGRAM FACTORIZATION
Shogo Seki1) , Hirokazu Kameoka2) , Tomoki Toda3) , Kazuya Takeda4)
1)2)

Graduate School of Informatics, Nagoya University
NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation
3)
Information Technology Center, Nagoya University
4)
Institutes of Innovation for Future Society, Nagoya University
ABSTRACT

While time-frequency masking is a powerful approach for speech enhancement in terms of signal recovery accuracy (e.g., signal-to-noise
ratio), it can over-suppress and damage speech components, leading to limited performance of succeeding speech processing systems.
To overcome this shortcoming, this paper proposes a method to restore missing components of time-frequency masked speech spectrograms based on direct estimation of a time domain signal. The
proposed method allows us to take account of the local interdependencies of the elements of the complex spectrogram derived fromthe redundancy of a time-frequency representation as well as theglobal structure of the magnitude spectrogram. The effectivenessof the proposed method is demonstrated through experimental evaluation, using spectrograms ﬁltered with masks to enhance of noisy
speech. Experimental results show that the proposed method significantly outperformed conventional methods, and has the potential toestimate both phase and magnitude spectra simultaneously and precisely.
Index Terms— Missing component restoration, Time-domain
spectrogram factorization
1. INTRODUCTION
The presence of background noise can signiﬁcantly degrade the quality of speech traveling through transmission systems and negatively
affect the performance of speech recognition and speech conversion
systems. The performance of these systems can be improved by suppressing the noise in observed audio signals and enhancing the target
speech.
One effective approach for speech enhancement involves timefrequency masking, which extracts only the components in thetime-frequency slots that are expected to be dominated by the target speech [1]. There are several ways to perform time-frequency
masking. For example, by using microphone inputs we can cluster time-frequency slots according to the direction of arrival ofeach source [2]. For monaural recording, we can use deep neuralnetworks to assign a source label to every time-frequency slot, or
partition the spectrogram into different source regions according to
the local “texture” of the spectrogram [3, 4]. While these methods
allow aggressive suppression of noise components, they can also
over-suppress the speech component and damage its acoustic features. As a result, the performance of speech processing systems canbe limited, even if a high signal-to-noise ratio (SNR) is obtained.
c
978-1-5090-6341-3/17/$31.00 2017
IEEE

To overcome this limitation, this paper deals with the problem ofrestoring the missing components of over-masked spectrograms.
One conventional missing component restoration approach formasked spectrograms is based on Non-negative Matrix Factorization (NMF) [5, 6]. NMF-based methods attempt to restore missing
components by assuming that the entire spectrogram can be approximated as a low-rank matrix, namely, as the product of two nonnegative matrices [7]. Modeling the entire spectrogram in this way
amounts to assuming that the magnitude spectrum observed at eachtime frame can be approximated as the sum of a limited numberof spectral templates. The signal can then be reconstructed usinga phase reconstruction algorithm [8]. Since spectrograms are generally redundant representations of time-domain signals, the magnitude and phase of each time-frequency slot are in fact interdependent
on each other. In other words, spectrograms must satisfy a certain
constraint in order to be associated with time-domain signals. [8]uses this fact as the basis for devising a phase reconstruction algorithm. This implies that we can also use this relationship as a clue tohelp restore the missing components of masked spectrograms. However, the performance of common phase reconstruction methods isstill insufﬁcient, and NMF-based methods also require some priorinformation about the target speech to be effective.
Recently, a time-domain extension of NMF called Time-domain
Spectrogram Factorization (TSF) has been proposed [9]. As thename implies, TSF performs NMF-like signal decomposition in thetime domain by taking account of the intrinsically redundant structure of spectrograms. While regular NMF approximates an observed
magnitude spectrogram into the sum of rank-1 spectrograms, TSF
decomposes an observed time-domain signal into the sum of L signal components, such that the magnitude spectrogram of each component is as close to a rank-1 structure as possible. In this work, we
propose using TSF to directly estimate the waveform signal such thatits magnitude spectrogram can be approximated as a low-rank matrix so that missing component restoration and phase reconstruction
can be performed jointly in a principled manner.
Cepstral Distance Regularization (CDR) is a recently proposed
technique used in semi-supervised NMF (SSNMF), which aims toenhance target speech in both the spectral and cepstral domains [10].
CDR does this by optimizing a combined objective function composed of an NMF-based model ﬁtting criterion deﬁned in the spectral
domain and a Gaussian mixture model-based probability distribution
deﬁned in the Mel-Frequency Cepstral Coefﬁcient (MFCC) domain.
This paper proposes a TSF-based missing component restoration method which combines the conventional methods discussed
above in a novel manner. The proposed method considers; 1) cues

of local dependencies of each component, which are detected using
redundancy in time-frequency domain expression, and/or 2) prior information about the target speech in a feature space, in addition tocues considered by conventional NMF-based methods. The effectiveness of the proposed method is demonstrated through the experimental restoration of masked speech spectrograms which are obtained by applying Ideal Binary Mask (IBM) ﬁlters to noisy speech.
The restoration performance of the proposed TSF-based method isthen compared with that of conventional NMF-based methods.
2. PRELIMINARIES
2.1. Non-negative Matrix Factorization (NMF)
NMF can be used to approximate an observed magnitude spectrogram, interpreted as a non-negative matrix X ∈ RK×M
, as a low≥0
rank matrix by factorizing X into the product of two non-negative
matrices H ∈ RK×Land U ∈ RL×M:
≥0≥0
X ≈ HU .

(1)

This amounts to assuming that the magnitude spectrum observed ateach time frame can be approximated as the sum of L basis spectra:
Xk,m  X̂k,m =

Hk,l Ul,m .

(2)

l2.2. Time-domain Spectrogram Factorization (TSF)
TSF is a novel signal decomposition technique that aims to directly
decompose an observed time-domain signal s ∈ RN (where N denotes the number of the samples of the entire signal) into the sum of
L signal components:


sl ,

(3)

l

such that the magnitude spectrogram of sl becomes as close to arank-1 (or low-rank) structure as possible. This idea can be formulated as an optimization problem of minimizing:I(θ) =

lsubject to2(|ψ H
k,m sl | − Hk,l Ul,m ) + R(U ),(4)sl = s,

(5)

k,m



Although speech enhancement methods based on semi-supervised
NMF are powerful in terms of signal recovery accuracy (e.g., signalto-noise ratio), they do not necessarily lead to an improvement in thequality of the enhanced speech in the feature domain [11]. To overcome this limitation, CDR has recently proposed to jointly enhancespeech both in the spectral and cepstral domains [10]. CDR forces
the estimated spectrogram to follow the same statistical distribution
as the training data in the feature space by introducing a regularization term given as the negative logarithm of the Gaussian mixture
density:
 


 
2
−K X̂ = − log, (6)wp
N X̂q,m ; μp,q , σp,qX̂q,m =

mpcq,r logrq

fr,k X̂k,m ,(7)kwhere X̂ ∈ RQ×M is a sequence of MFCCs of X̂. f = {fr,k }r,k ∈RR×K and c = {cq,r }q,r ∈ RQ×R represent a mel-ﬁlterbank coefﬁcient matrix and an inverse discrete cosine transform matrix, respectively. Eq. (6) represents the negative log-likelihood of a Gaussian Mixture Model (GMM) with parameters {wp , μp , Σp }p . Notethat the GMM parameters are pretrained using training examples ofclean speech.
3. PROPOSED METHOD

If the magnitude spectrogram of an audio signal of interest canbe assumed to have a low-rank structure, missing components in themagnitude spectrogram can be restored by ﬁtting the NMF model (1)over the observable regions [7]. The time-domain signal can thenbe synthesized for example by using a phase reconstruction technique [8].

s=2.3. Cepstral Distance Regularizationlwhere R(U ) is a sparse regularization term. ψ Hk,m sl represents thetime-frequency element of sl (i.e., a Short-Time Fourier Transform(STFT), or Wavelet Transformation), and ψ k,m ∈ CN is a complex
sinusoid windowed at time frame tm with center frequency ωk .
Since this method allows to directly estimate the signal components s1 , . . . , sL in the time domain, the phase reconstruction procedure is implicitly involved in the spectrogram factorization process.
This gives TSF its name.

3.1. Problem Setting
Let Y ∈ CK×M be an observed complex spectrogram with missing
components whose components are represented as Yk,m and wherek ∈ {1, . . . , K} and m ∈ {1, . . . , M } are indices of frequency
bins, and time frames, respectively. By using Γ to denote the set ofthe observable time-frequency slots of Y , here we assume that the
STFT coefﬁcients in the missing regions are zero:Yk,m = 0,((k, m) ∈
/ Γ).

(8)

We would like to estimate these components so that the time-domain
signal can be reconstructed.
3.2. Objective Function Design
We can use the TSF framework to impute missing components byconsidering local interdependencies of the elements of a complex
spectrogram. We can also borrow the idea from the conventional
NMF-based approach to estimate the magnitude part of the missing
components by assuming the magnitude spectrogram to have a lowrank structure. Additionally, we use CDR to ensure that the restored
spectrogram follows a pretrained distribution in the cepstral domain.
Hence, we propose introducing the following objective function to
be minimized:2|ψ HI(θ) =
k,m s − Yk,m |(k,m)∈Γ

+ λ1





D· |ψ Hk,m s| | X̂k,m

k,m

+ λ2





D· |Yk,m | | X̂k,m(k,m)∈Γ − λ3 K X̂ ,(9)

4.2. Update rules for s
In Eq. (9), the ﬁrst and second terms are related to parameter s.
When D· is deﬁned as the squared Euclidean distance, as in [9],we can show:
DEU |ψ H
k,m s| | X̂k,m

k,m

=



2
H
2
|ψ H
k,m s| − 2|ψ k,m s|X̂k,m + X̂k,m



k,m

≤



2
|ψ H
k,m s − X̂k,m ak,m | .

(11)

k,m

Here, we can use the right-hand side of this inequality as a majorizer
for the second term of Eq. (9), where a = {ak,m }k,m is an auxiliary
parameter. The equality holds when:

Fig. 1. Illustration of the designed objective function
where λ1 , λ2 and λ3 are hyperparameters that weigh the importanceof the second, third and fourth terms, respectively, θ = {s, H, U }is the set of parameters to be optimized, and D· is a divergence measure between non-negative arguments. In this paper, either squared
Euclidean distance or Kullback-Leibler (KL) divergence are used.
Fig. 1 shows an overview of the proposed method, where the circled
numbers correspond the individual terms of the objective function.
In Eq. (9), the ﬁrst term represents the squared error between an
observed complex spectrogram and that of the estimated signal sover the observable regions Γ. It is important to note that ψ Hk,m s
always satisﬁes the condition that all complex spectrograms mustsatisfy and thus the redundancy of the time-frequency representation
is implicitly considered. The second term represents the error between Eq. (2) and the magnitude spectrogram of s, which connects
the ﬁrst and the third term effects. The third term represents the error between Eq. (2) and the observed magnitude spectrogram. This
term corresponds to the objective function of the conventional NMFbased approach. The fourth term is cepstral distance regularization
term given in Eq. (6). This forces X̂ to follow the statistical distribution of target speech in the feature space domain. By optimizing theobjective function, the missing components of observed spectrogram
Y can be restored by satisfying the provided constraints.

ak,m =

4.1. Majorization-minimization principle
We use F (θ) to denote an objective function that we want to minimize with respect to θ. F + (θ, α) is deﬁned as a ”majorizer” forF (θ) if it satisﬁes:
F (θ) = min F + (θ, α).
α(10)We call α an auxiliary variable. By using F + (θ), F (θ) can be iteratively decreased according to the following theorem:
Lemma 1
F (θ) is non-increasing under the updates, θ ← argminθ F + (θ, α)and α ← argminα F + (θ, α).

|ψ Hk,m s|.

(12)

When D· is deﬁned as the KL-divergence, we can show:

k,m

=


k,m

≤



DKL |ψ H
k,m s| | X̂k,m




|ψ H
k,m s| log|ψ Hk,m s|
X̂k,m− |ψ Hk,m s| + X̂k,m




2
∗
H
Fk,m |ψ H
k,m s| − 2Re Gk,m ψ k,m s + X̂k,m + const.,

k,m

(13)
where Dk,m , Fk,m , and Gk,m are given by:
Dk,m = log

ξk,m
− 2,
X̂k,m

Fk,m =

Dk,m1+ ξk,m
,
2bk,m
1
,
ξk,m

Gk,m =

0,−Dk,m a∗k,m /2,

4. PARAMETER ESTIMATION ALGORITHM
Here we describe a novel convergence-guaranteed algorithm forminimizing Eq. (9) based on a majorization-minimization (MM)principle.

ψH
k,m s

(14)
(Dk,m ≥ 0)

,

(15)

(Dk,m ≥ 0).
(Dk,m < 0)

(16)

(Dk,m < 0)

Similarly, we can use the right-hand side of this inequality as a majorizer for the case of the KL-divegence where b = {bk,m }k,m , andξ = {ξk,m }k,m are auxiliary parameters. The equality of Eq. (13)satisﬁes when:bk,m = ξk,m = |ψ H
k,m s|.

(17)

Since both Eqs. (11) and (13) are differentiable and convex, an
optimal update for s minimizing Eq. (11) or Eq. (13) can be foundusing gradient methods. In the case of the squared Euclidean distance, parameter s can be efﬁciently updated in the following way.
For the ﬁrst term of Eq. (9), let Ỹk,m deﬁned as follows:Ỹk,m =

Yk,m ,
Sk,m ,((k, m) ∈ Γ),((k, m) ∈ Γ)

(18)

SinceH2(k,m)∈Γ |ψ k,m s − Sk,m | ≥ 0, we obtain:



2
|ψ H
k,m s − Yk,m |(k,m)∈Γ

≤



2
|ψ H
k,m s − Yk,m | +(k,m)∈Γ=

where β = {βk,l,m }k,l,m satisﬁes:
βk,l,m =2|ψ Hk,m s − Sk,m |(k,m)∈Γ2|ψ Hk,m s − Ỹk,m | .

(19)

k,m

Thus, we can also use the right-hand side of Eq. (19) as a majorizer
where S = {Sk,m }k,m is an additional set of auxiliary parameters.
The equality of Eq. (18) holds when:
Sk,m = ψ Hk,m s.

(20)

Since this majorizer is given as a quadratic function of s, obtain an
update rule for s analytically as follows:
⎞−1
⎛
1 ⎝
⎠
Re[ψ k,m ψ Hs=k,m ]1 + λ1
k,m
⎛
⎞

(21)
×⎝
Re[ψ k,m (Ỹk,m + λ1 X̂k,m ak,m )]⎠ .
k,mAlthough Eq. (21) contains inverse matrix computation, this can be
avoided by selecting ψ k,m , so that k,m ψ k,m ψ Hk,m becomes a circulant matrix. It can be diagonalized using discrete Fourier transH
form matrix F as follows: k,m Re[ψ k,m ψ Hk,m ] = F V F . The
inverse matrix can now be calculated efﬁciently. For example, when
ψ k,m represents an STFT with a square-root Hanning window, diagonal matrix V becomes an identity matrix.
When D· is deﬁned as the a KL-divergence, the inverse matrix
computation is unavoidable because the matrix to be inverted does
not become a circulant matrix. Instead of trying to obtain an update
rule with an analytical form, here we choose to update s be obtainedusing a gradient method where the gradient is given in the form:Re[ψ k,m {(rk,m + λ1 Fk,m )ψ H
∇s I(θ) =2
k,m s
k,m

− (rk,m Yk,m + λ1 Gk,m )}].

(22)

where ak,l , bk,l , ck,l , dl,m , el,m , and fl,m are deﬁned as follows:
ak,l =

(λ1 + λ2 rk,m )Ul,m

m

+ λ3

Ar,m p(ζr,m ) +r,mbk,l = −


m

− λ3

δBr,m ≥0 |Br,m |
φr,m
fr,k Ul,m ,(29)(λ1 |ψ H
k,m s| + λ2 rk,m |Yk,m |)βk,l,m



δBr,m <0 |Br,m |vr,k,l,m ,

(30)

r,m

ck,l = − λ3dl,m =





Ar,m

r,m

ρ2r,k,l,m,
fr,k Ul,m(λ1 + λ2 rk,m )Hk,l

k

+ λ3



Ar,m p(ζr,m ) +r,kel,m = −k− λ3
fl,m = − λ3

1, ((k, m) ∈ Γ).
0, ((k, m) ∈
/ Γ)

(23)

4.3. Update rules for H and U

δBr,m ≥0 |Br,m |
φr,m
fr,k Hk,l ,(31)(λ1 |ψ H
k,m s| + λ2 rk,m |Yk,m |)βk,l,m



δBr,m <0 |Br,m |vr,k,l,m ,

(32)



ρ2r,k,l,m.
fr,k Hk,l

r,k

(33)

Ar,m

δx is an indicator function which takes the value of one when condition x is satisﬁed, otherwise its value is zero. Note that Lr,m , Ar,m ,Br,m , and p(·) are deﬁned as follows:
Lr,m =In Eq. (9), the second, third and fourth terms are related to parameters H, and U . When D· is deﬁned as the squared Euclidean distance, the update rules for H, and U can be obtained in the samemanner as the regular NMF, as follows:Hm (λ1 |ψ k,m s| + λ2 rk,m |Yk,m |)Ul,m,2
Ul,mm (λ1 + λ2 rk,m ) β(24)Hk (λ1 |ψ k,m s| + λ2 rk,m |Yk,m |)Hk,l2
Hk,lk (λ1 + λ2 rk,m ) βk,l,m

,

(25)



fr,k X̂k,m ,

(34)

k



ηp,m c2q,r
,
2 ω
2σp,q
p,q,r,mp,q ηp,m cq,r ϕp,q,r,m
Br,m = −,2 ωσp,qp,q,r,mp,qAr,m =p(ζr,m ) =k,l,m

Ul,m =

r,kNote that terms with the form k,m Re[ψ k,m ·] can be computed
efﬁciently using the Fast Fourier Transform (FFT).

Hk,l =(26)

When D· is deﬁned as the KL-divergence, the update rules for
H and U can be derived as in [10], as follows:−bk,l + b2k,l − 4ak,l ck,l,(27)
Hk,l =
2ak,l
−el,m + e2l,m − 4dl,m fl,mUl,m =
,
(28)
2dl,m

Here, rk,m is binary variable deﬁned as:
rk,m =

Hk,l Ul,m.
X̂k,m2 log ζr,m
1
+ 2 .
ζr,m
ζr,m

(35)
(36)
(37)

η = {ηp,m }p,m , ϕ = {ϕp,q,r,m }p,q,r,m , ρ = {ρr,k,l,m }r,k,l,m ,v = {vr,k,l,m }r,k,l,m , ζ = {ζr,m }r,m , and φ = {φr,m }r,m are allζr,m = φr,m = Lr,m ,

(39)
(40)

10
0
-10

(41)

-20

and ω = {ωp,q,r,m }p,q,r,m is an arbitrary positive constant parameter satisfying r ωp,q,r,m = 1.

-30

Unprocessed
EU-NMF
KL-NMF
EU-TSF
KL-TSF
KL-TSF w/ Reg.

-15

-10

-5

0

8
6
4
2

5

10

0

15

-15

-10

-5

0

5

10

15

Target-to-masking ratio [dB]

Target-to-masking ratio [dB](a) SNR(b) MFCC distance

5. EXPERIMENTAL EVALUATIONS

Fig. 2. Results for TMR dataset

5.1. Settings

12

10
0
-10
Unprocessed
EU-NMF
KL-NMF
EU-TSF
KL-TSF
KL-TSF w/ Reg.

-20
-30
-15

-10

-5

0

5

10

15

MFCC distance [dB]

10

(N)

where Sk,m and Sk,m are complex spectrograms of clean speechand noise, respectively, and  is the threshold determining whether
each component activates or not. As clean data, 200 utterances of
20 speakers from ATR 503 database, including males and females
were used [12]. Babble noise was added to each clean speech sample at various SNR or threshold settings in Eq. (42) while building the IBMs for noisy speech, and then the spectrograms of thenoisy speech were masked. Three datasets were built for the experiments; Target-to-masking ratio dataset (TMR dataset) in which
noisy speech were made under varying SNR conditions, while thecorresponding IBMs were constructed at a ﬁxed threshold parameter(0 dB), Target-to-masking threshold dataset (TMT dataset) in which
noisy speech were made at a ﬁxed SNR setting (0 dB), but the corresponding IBMs were constructed at varying thresholds. and a Overmasking dataset in which noisy speech were made at a ﬁxed SNR
setting (0 dB), and the corresponding IBMs were constructed at aﬁxed threshold parameter (0 dB), however, the existing components
through IBM ﬁltering were moreover erased by making them zeros
randomly and compulsorily, and it was considered in more practicalconditions.
Three TSF-based methods were investigated as proposed methods: a TSF using the squared Euclidean distance (EU-TSF), andthe TSFs using KL-divergence with or without cepstral distanceregularization (KL-TSF w/ Reg., KL-TSF w/o Reg). Two NMFbased methods using the squared Euclidean distance (EU-NMF) andthe KL-divergence (KL-NMF) were used to represent conventional
methods. Each speech signal was sampled at 16 kHz, and the spectrograms were obtained through frame analysis using 32 ms and 16
ms shifts with square-root Hanning windows. The total number ofbasis spectra was set to 30, and the total number of iterations forparameter updating was 200. Weight parameters λ1 , λ2 , λ3 were
adjusted during the ﬁrst half of the iterations so that each term of
the objective function in Eq. (9) had the same magnitude, and theseweight parameters were then ﬁxed during the second half of theiterations. For cepstral distance regularization, 0-to-13th MFCCs

Unprocessed
EU-NMF
KL-NMF
EU-TSF
KL-TSF
KL-TSF w/ Reg.

20

SNR [dB]

The performance of the proposed methods was evaluated through experiments with speech spectrograms which had been masked using
IBMs for noise elimination. The masked spectrograms were prepared using IBMs constructed of clean speech and noise data, which
can be represented as:


⎧
(C) 2
|
|S
⎨1,
10 log10 k,m>(N)|Sk,m |2,
(42)
MIBM =⎩0, (otherwise)(C)

Unprocessed
EU-NMF
KL-NMF
EU-TSF
KL-TSF
KL-TSF w/ Reg.

10

(38)

MFCC distance [dB]

ϕp,q,r,m = cq,r log Lr,m + ωp,q,r,m (μp,q − Xq,m ),fr,k Hk,l Ul,m,
ρr,k,l,m = vr,k,l,m = k ,l fr,k Hk ,l Ul ,m12
20

SNR [dB]

auxiliary parameters satisfying following relations:2
wp q N (Xq,m ; μp,q , σp,q)
ηp,m =
,
2




w
N
(X
;
μ
,
σ


p
q ,m
p ,q
p
q
p ,q  )

8
6
4
2
0

-15

-10

-5

0

5

10

15

Target-to-masking threshold [dB]

Target-to-masking threshold [dB](a) SNR(b) MFCC distanceFig. 3. Results for TMT datasetwith 20-dimensional mel-ﬁlterbanks were extracted from 100 otherutterances of the individual speakers and were for GMM training.
The mixture component of the GMM was set to 30. For KL-TSF,
the Adadelta technique was used for gradient descent for s [13]. A
phase spectrogram for the NMF-based methods was reconstructedusing the Grifﬁn-Lim algorithm with 100 iterations [8]. As measurements of performance, SNRs and the MFCC distances between
the restored speech and the corresponding clean speech were used.
In addition, restoration of each masked spectrogram was repeated
3 times owing to the weakening effect of the initial values, andmeasurements were averaged over all of the iterations and speech.
5.2. Results
Figs. 2–4 show the SNR and the MFCC distance results for TMR
dataset, TMT dataset, and Over-masking dataset. In Fig. 2, the horizontal axis shows SNR settings, while in Fig. 3 it represents thethreshold settings of the IBMs. In Fig. 4, the horizontal axis shows
missing rate for existing components of IBMs. The vertical axesin these ﬁgures represent performance. Error bars in the ﬁgures
represent 95 % conﬁdence intervals. Unprocessed results, whose
waveform signals were obtained by reproducing the masked spectrograms straightforwardly without any reconstruction methods, arealso shown in each ﬁgure.
These results show that the proposed TSF-based methods outperformed conventional NMF-based methods. Especially, we cansee that the SNRs of TSF-based methods follow similar tendencies

12

5

10

SNR [dB]

0-5-10

Unprocessed
EU-NMF
KL-NMF
EU-TSF
KL-TSF
KL-TSF w/ Reg.

-15
-20

0

15

30

45

60

75

90

MFCC distance [dB]10

8. REFERENCES
Unprocessed
EU-NMF
KL-NMF
EU-TSF
KL-TSF
KL-TSF w/ Reg.

8[1] Nathalie Virag, “Single channel speech enhancement basedon masking properties of the human auditory system,” IEEE
Transactions on speech and audio processing, vol. 7, no. 2, pp.
126–137, 1999.

6[2] Ozgur Yilmaz and Scott Rickard, “Blind separation of speech
mixtures via time-frequency masking,” IEEE Transactions on
Signal Pprocessing, vol. 52, no. 7, pp. 1830–1847, 2004.

4
2
0

0

15

30

45

60

75

Missing rate [%]Missing rate [%](a) SNR(b) MFCC distance90

Fig. 4. Results for Over-masking dataset
to that of the unprocessed result unlike NMF-based methods. Thisis because that the NMF-based methods estimate not missing correct
phase information, but estimate consistent phase information for thereconstructed spectrogram, which leads insufﬁcient performance.
Moreover, the TSF-based methods maintained high SNRs similar to Unprocessed while greatly improving the MFCC distances.
The proposed method using the squared Euclidean distance (EUTSF) delivered especially stable results. In the over-suppress conditions (Fig. 4), we can see that the proposed methods using either thesquared Euclidean distance (EU-TSF) or the KL-divergence (KLTSF) exceed unprocessed results both in the SNRs and the MFCC
distances. This suggests that the proposed methods are potential torestore the missing components well by applying over-masked spectrograms, and especially KL-TSF could achieve quite robust performance for the spectrograms.
These experiments also show that cepstral distance regularization does not consistently improve restoration performance. Thisis because the intensity of regularization can be unsuitable and canactually degrade performance. The balancing of error functions andregularization terms has not been sufﬁciently researched and remainsa challenging problem.

6. CONCLUSION
This paper proposed a novel missing component restoration method
for masked speech spectrograms based on a TSF signal decomposition model. The proposed method attempts to utilize as many acoustical cues as possible, e.g., cues observed in spectrograms as wellas cues from spectrograms of target speech in a feature space, and,if possible, to directly estimate waveform signals. The experimental results showed that the proposed TSF-based restoration significantly outperform conventional NMF-based methods, and has potential to estimate both magnitude and phase spectra simultaneously
and precisely. These results also demonstrated that a part of the TSFbased restoration methods has quite robust performance for oversuppressing occurring in time-frequency masking.

7. ACKNOWLEDGMENT
This work was supported in part by JSPS KAKENHI Grant Number
17H01763.

[3] Yuxuan Wang and DeLiang Wang, “Towards scaling up
classiﬁcation-based speech separation,” IEEE Transactions on
Audio, Speech, and Language Processing, vol. 21, no. 7, pp.
1381–1390, 2013.
[4] John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji
Watanabe, “Deep clustering: Discriminative embeddings forsegmentation and separation,” in Acoustics, Speech and Signal
Processing (ICASSP), 2016 IEEE International Conference on.
IEEE, 2016, pp. 31–35.
[5] Daniel D Lee and H Sebastian Seung, “Algorithms for nonnegative matrix factorization,” in Advances in Neural Information Processing Systems, 2001, pp. 556–562.
[6] Paris Smaragdis and Judith C Brown, “Non-negative matrix
factorization for polyphonic music transcription,” in Applications of Signal Processing to Audio and Acoustics, 2003 IEEE
Workshop on. IEEE, 2003, pp. 177–180.
[7] Paris Smaragdis, Bhiksha Raj, and Madhusudana Shashanka,“Missing data imputation for time-frequency representationsof audio signals,” Journal of Signal Processing Systems, vol.
65, no. 3, pp. 361–370, 2011.
[8] Daniel Grifﬁn and Jae Lim, “Signal estimation from modiﬁed
short-time fourier transform,” IEEE Transactions on Acoustics,
Speech, and Signal Processing, vol. 32, no. 2, pp. 236–243,
1984.
[9] Hirokazu Kameoka, “Multi-resolution signal decomposition
with time-domain spectrogram factorization,” in Acoustics,
Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015, pp. 86–90.
[10] Li Li, Hirokazu Kameoka, Takuya Higuchi, and Hiroshi
Saruwatari, “Semi-supervised joint enhancement of spectral
and cepstral sequences of noisy speech,” Interspeech 2016, pp.
3753–3757, 2016.
[11] Paris Smaragdis, Bhiksha Raj, and Madhusudana Shashanka,“Supervised and semi-supervised separation of sounds fromsingle-channel mixtures,” Independent Component Analysis
and Signal Separation, pp. 414–421, 2007.
[12] Akira Kurematsu, Kazuya Takeda, Yoshinori Sagisaka,
Shigeru Katagiri, Hisao Kuwabara, and Kiyohiro Shikano,“ATR Japanese speech database as a tool of speech recognition and synthesis,” Speech Communication, vol. 9, no. 4, pp.
357–363, 1990.
[13] Matthew D Zeiler, “Adadelta: an adaptive learning rate
method,” arXiv preprint arXiv:1212.5701, 2012.