Consistent Wiener Filtering:
Generalized Time-Frequency Masking
Respecting Spectrogram Consistency
Jonathan Le Roux1 , Emmanuel Vincent2 , Yuu Mizuno3 ,Hirokazu Kameoka1 , Nobutaka Ono3 , and Shigeki Sagayama3
1

NTT Communication Science Laboratories, NTT Corporation,
3-1 Morinosato Wakamiya, Atsugi, Kanagawa 243-0198, Japan
2
INRIA, Centre Inria Rennes - Bretagne Atlantique,
Campus de Beaulieu, 35042 Rennes Cedex, France
3
Graduate School of Information Science and Technology, The University of Tokyo,
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan{leroux,kameoka}@cs.brl.ntt.co.jp, emmanuel.vincent@inria.fr,{mizuno,onono,sagayama}@hil.t.u-tokyo.ac.jp
Abstract. Wiener ﬁltering is one of the most widely used methods inaudio source separation. It is often applied on time-frequency representations of signals, such as the short-time Fourier transform (STFT), to
exploit their short-term stationarity, but so far the design of the Wiener
time-frequency mask did not take into account the necessity for the output spectrograms to be consistent, i.e., to correspond to the STFT of atime-domain signal. In this paper, we generalize the concept of Wiener
ﬁltering to time-frequency masks which can involve manipulation of thephase as well by formulating the problem as a consistency-constrained
Maximum-Likelihood one. We present two methods to solve the problem, one looking for the optimal time-domain signal, the other promoting consistency through a penalty function directly in the time-frequency
domain. We show through experimental evaluation that, both in oracle
conditions and combined with spectral subtraction, our method outperforms classical Wiener ﬁltering.
Keywords: Wiener ﬁltering, Short-time Fourier transform, Spectrogram
consistency, Source separation, Spectral subtraction.

1

Introduction

Wiener ﬁltering has been one of the most widely used methods for source separation for several decades, in particular in audio signal processing. To exploit the
short-term stationarity of audio signals, it is very often applied on time-frequency
representations [1], especially the short-time Fourier transform (STFT). However, classical Wiener ﬁltering does not take into account the intrinsically redundant structure of STFT spectrograms, and its output is actually in general not
the optimal solution. We show here that by ensuring that the output spectrograms are “consistent”, i.e., that they correspond to actual time-domain signals,
we can obtain a more eﬃcient ﬁltering. Many of the most promising methods for
V. Vigneron et al. (Eds.): LVA/ICA 2010, LNCS 6365, pp. 89–96, 2010.
c Springer-Verlag Berlin Heidelberg 2010


90

J. Le Roux et al.

source separation exploit spectral models of the sources (non-negative matrix
factorization, Gaussian mixture models, autoregressive modeling, etc.) and, as
these models are often based on Gaussian assumptions, they commonly involve
Wiener ﬁltering as a post-processing [2]. It is thus of tremendous importance toensure that the information gathered by these algorithms is best exploited.
Wiener ﬁltering can be formulated as the solution of a Maximum-Likelihood(ML) problem in the time-frequency domain without constraint on the space ofadmissible solutions. The classical solution then only involves a manipulationon the magnitude part of the spectrograms, leading in general to arrays of complex numbers which do not correspond to any time-domain signal. We generalize
here the concept of Wiener ﬁltering to time-frequency masks which can involve a
manipulation of the phase as well in order to ﬁnd the ML solution among consistent spectrograms. Formulating the problem as the minimization of an objectivefunction derived from the Wiener likelihood and explicitely taking into accountconsistency, we present two methods to solve it: one consists in computing theexact optimum by solving the problem in the time domain; the other relies on arelaxation of the consistency constraints through the introduction of a penalty
function promoting consistency of the output spectrogram. We already exploitedthe idea of consistency-promoting penalty functions for fast signal reconstruction from modiﬁed magnitude spectrograms [3] and to improve the modelingaccuracy of the complex non-negative matrix factorization framework [4]. It enables us here to develop an eﬃcient algorithm which computes an approximate
solution close to the true optimum obtained with the time-domain method.
We evaluate the performance of the proposed methods compared to classicalWiener ﬁltering on two tasks: separation of concurrent speech by two speakers
under oracle conditions, and denoising of speech mixed with synthetic and realworld background noises where only the noise mean power spectra are known
and the speech spectrum is estimated through spectral subtraction [5].

2

Wiener Filtering and Consistency

2.1

Maximum-Likelihood Formulation

We assume that the observed signal x is the mixture of two signals, a target s1
and an interference signal s2 . We further assume that the STFT coeﬃcients S1and S2 of the signals s1 and s2 at each time frame t and frequency bin ω aremodeled as statistically independent Gaussian random variables with variance
σ12 and σ22 respectively. For convenience of notation, we shall write ν (i) = 1/σi2 .
Note that the case of several interference signals can be reduced, without lossof generality, to that of two sources only, as we assume in particular that thesources are not correlated.
Denoting by X the spectrogram of x, classical Wiener ﬁltering consists inmaximizing the log-likelihood of the STFT coeﬃcients S1 and S2 , which can be
written, under the constraint that X = S1 +S2 , as a function of S = S1 only: (2)
1   (1)
L(S) = −νω,t |Sω,t |2 +νω,t |Xω,t − Sω,t |2 + C(ν (1) , ν (2) ) ,
(1)
2 ω,t
ω,t

Consistent Wiener Filtering

91

where C is a constant depending only on ν (1) , ν (2) . Introducing the classicalWiener ﬁltering estimate for S1 ,(2)Ŝω,t =

νω,t
(1)

(2)

νω,t + νω,tXω,t ,(2)

the ML problem can be reformulated as the minimization of the objective function

(1)
(2)
ψ(S) =
αω,t |Sω,t − Ŝω,t |2 , where αω,t = νω,t + νω,t .
(3)
ω,t

2.2

Wiener Filtering with Consistency Constraint

If no further constraint is assumed on S, the objective function is obviously minimized for S = Ŝ. However, we need to keep in mind that the STFT is a redundant representation with a particular structure. Denoting by N the number of
frequency bins and T the number of frames, STFT spectrograms of time-domain
signals are elements of CN T , which we shall call “consistent spectrograms”, but
one of the fundamental points of this paper is that not all elements of CN T canbe obtained as such [6,3]. If we assume that inverse STFT is performed in sucha way that a signal can be exactly reconstructed from its spectrogram throughinverse STFT, then we showed in [3] that a necessary and suﬃcient condition
for an array W to be a consistent spectrogram is for it to be equal to the STFT
of its inverse STFT. The set of consistent spectrograms can thus be described as
the null space Ker(F ) of the R-linear operator F from CN T to itself deﬁned by
F (W ) = G(W ) − W, where G(W ) = STFT(iSTFT(W )) .

(4)

Going back to the Wiener ﬁltering problem, if we now impose that the solution
be consistent, the problem amounts to ﬁnding a consistent spectrogram S minimizing ψ, or in other words to minimize ψ under the constraint that F (S) = 0.
Imposing consistency is not a mere elegance or theory-oriented concern, but atruly fundamental problem. Indeed, the spectrogram of the signal resynthesizedfrom the classical Wiener ﬁlter spectrogram Ŝ is actually diﬀerent in generalfrom Ŝ, and no longer maximizing the Wiener log-likelihood (or minimizing ψ),
so that the ﬁnal result of the processing that we are listening to is in fact not
the optimal solution. What we really want to do is to ﬁnd a signal such thatits spectrogram minimizes the Wiener criterion ψ, or, formulating this in thetime-frequency domain, to minimize the following “true” objective function2αω,t G(S)ω,t − Ŝω,t  ,(5)ψ̃(S) =
ω,twhere G(S) is again the spectrogram of the signal resynthesized from S by inverse STFT. We can try to solve the problem directly in the time domain byminimizing ψ(STFT(s)) w.r.t. the time-domain signal s. Another possibility is
to relax the consistency constraint by introducing it as a penalty function: if
the weight of the penalty is chosen suﬃciently large, or is increased during thecourse of the optimization, the estimated spectrogram should ﬁnally be bothconsistent and minimizing ψ among the consistent spectrograms.

92

J. Le Roux et al.

3

Optimization Algorithms

3.1

Time-Domain FormulationTheﬁltering optimization problem amounts to minimizing
 consistent Wiener
2
α
|S
−
Ŝ
|
on
the subspace of consistent spectrograms, while that
ω,t
ω,t
ω,t
ω,t
of estimating the signal whose STFT spectrogramis closest to the modiﬁed
STFT spectrogram Ŝ amounts to minimizing ω,t |Sω,t − Ŝω,t |2 on the same
subspace [6]. The latter problem can be transformed through Parseval’s theorem
into the minimization of a simple quadratic form on the time signal parameters,
but the weights α make here the computation of the optimal signal cumbersome
as they hinder us from simplifying the product of the Fourier matrix and its
transpose. Let At be the N × N diagonal matrix with diagonal coeﬃcients αω,t ,F the N ×N Fourier transform matrix, wt the N ×L matrix which computes thet-th windowed frame of the signal x (of length L), and ŝt the inverse transformof the t-th STFT frame of Ŝ. We can show that the optimal signal x is given by H H−1  H Hx̂ =wt F At F wt
wt F At F ŝt .
(6)
t

t

If At were not present, as in the latter problem, then F H F would simplify
to
Id and weH would get the simple weighted overlap-add estimation x =
 NH
w
ŝ
/
t t t
t wt wt . However, the simpliﬁcation cannot be done here, leading toa very large (L × L) matrix inversion problem. Still, this matrix is band-diagonal(and Hermitian), and solving the system is possible in a reasonable amount oftime and using a reasonable amount of memory space. To reduce in particular
the memory requirements, we can split in practice the estimation of the signal
on overlapping blocks of a few frames, and reconstruct an approximate solution
on the whole interval by overlap-add from the locally optimal signals.
3.2

Consistency as a Penalty Function

For an array of complex numbers W ∈ CN T , F (W ) represents the relation
between W and the STFT of its inverse STFT. Instead of enforcing consistency
through the “hard” constraint F (W ) = 0, which may be diﬃcult to handle, we
can relax that constraint by using any vector norm of F (W ) to derive a numericalcriterion quantifying how far W is from being consistent. We consider here theL2 norm of F (W ), which leads, as shown in [3], to a criterion related to that usedby Griﬃn and Lim to derive their iterative STFT algorithm [6]. Introducing theconsistency penalty in (3), the new objective function to minimize reads
G(S)ω,t − Sω,t 2 .
(7)
ψγ (S) = ψ(S) + γ
ω,tAn eﬃcient optimization algorithm for ψγ can be derived through the auxiliary
function method [7]. A function ψγ+ (S, S̄) is called an auxiliary function for ψγ (S)and S̄ an auxiliary variable if ψγ (S) = minS̄ ψγ+ (S, S̄), ∀S. The minimization ofψγ can be performed indirectly by alternately minimizing ψγ+ w.r.t. S and S̄.
If we assume, as we shall do, that the inverse STFT is performed using thewindowed overlap-add procedure with the synthesis window before normalization

Consistent Wiener Filtering

93

equal to the analysis window, it results from [6] that G(S) is the closest consistent
spectrogram to S in a least-squares sense:




G(S)ω,t − Sω,t 2 = minS̄ω,t − Sω,t 2 , ∀S .
(8)
S̄∈Ker(F )

ω,t

ω,t

If we now deﬁne the function ψγ+ : CN T × Ker(F ) → R such thatSω,t − S̄ω,t 2 ,∀S ∈ CN T , ∀S̄ ∈ Ker(F ), ψγ+ (S, S̄) = ψ(S) + γ

(9)

ω,t

we easily see from (8) that ψγ+ is an auxiliary function for ψγ . This leads to an iterative optimization scheme in which, starting at step p from a spectrogram S (p) ,S̄ is ﬁrst updated to G(S (p) ), and the new estimate S (p+1) is simply estimatedas the minimum of a second-order form with diagonal coeﬃcients, altogether
resulting in the following update equation:
(p)

(p+1)

Sω,t

←

αω,t Ŝω,t + γG(Sω,t ).
αω,t + γ

4Experimental Evaluation

4.1

Settings and Implementations(10)

The sampling rate was 16 kHz. All spectrograms were built with a frame length
N = 1024, a frame shift R = 512 and a sine window for analysis and synthesis.
The time-domain method was implemented as follows: the analytical solution
is computed separately on blocks of 64 STFT frames; the blocks have a 50 %overlap, and the resulting short-time signals are cross-faded on a small region(here 16 frames) around the center of the overlap regions in order to discard
portions of signal near the block boundaries, expected to suﬀer from boundaryeﬀects. The above values for the block size and the amount of overlap and crossfade were determined experimentally so as to minimize computation and memory
costs while still obtaining solutions with a true Wiener criterion very close tothat of the analytical solution computed on the whole interval.
For the penalty-based algorithm, heuristically, the larger γ, the slower theconvergence, but the better the solution. We noticed experimentally that ψ̃monotonically decreased through the update (10) with γ ﬁxed when startingfrom a point obtained through updates with a smaller γ. We thus designed an
update scheme for γ: starting from a small value γ0 (typically 10−5 ) for γ, we
update S through (10) while slightly increasing γ by δ (initially set to γ0 as well)
until the decrease of ψ̃ becomes slower than 1 %, in which case we update δ to2δ and restart the process. We stop after two increases of δ without signiﬁcant
improvement of ψ̃, which typically occurred after around 200 iterations.
4.2

Speech Separation under Oracle Conditions

We evaluate here the performance of the proposed methods for the separation of
10 mixtures of two speakers under oracle conditions, i.e., assuming that the true

94

J. Le Roux et al.
Table 1. Performance comparison for speech separation under oracle conditions

MethodWiener
Griﬃn-LimTime domain
Penalty

SDR
15.0 dB
11.4 dB
17.1 dB
16.5 dB

ISR
25.0 dB
21.9 dB
28.2 dB
27.1 dB

SIR
24.6 dB
27.6 dB
27.5 dB
26.7 dB

SAR
15.6 dB
11.4 dB
17.7 dB
17.1 dBψ̃(S)2.0 × 109
6.8 × 1012
6.2 × 104
2.2 × 105Time
0.05 s18.1 s1423.0 s
8.1 s

power spectrograms of both sources are known. The speech signals were takenfrom the BSS Oracle Toolbox data [8], downsampled to 16 kHz and downmixed
to mono before being mixed together to obtain 12 s long 0 dB Signal to Distortion
Ratio (SDR) mixtures. For comparison, we also give the results for the classicalWiener ﬁlter output Ŝ (“Wiener”) and for the spectrogram whose magnitude isclosest to the magnitude of the classical Wiener ﬁlter, computed through Griﬃnand Lim’s iterative STFT algorithm [6] run for 400 iterations (“Griﬃn-Lim”).
This way of obtaining a consistent spectrogram through post-processing of theclassical Wiener ﬁlter magnitude seems indeed a natural method to attempt.
The results are summarized in Table 1. For each method are reported fourcommonly used objective source separation performance criteria [9], namely the
Signal to Distortion ratio, the source Image to Spatial distortion Ratio (ISR), theSignal to Interference Ratio (SIR) and the Signal to Artifacts Ratio (SAR), as
well as the computation time and the ﬁnal value of the “true” Wiener criterion ψ̃.
Although the performance of the classical Wiener ﬁlter is already very good, with
15.0 dB output SDR, we can see that the proposed methods all lead to signiﬁcant
improvements in both the true Wiener criterion ψ̃ and the objective performance
criteria, with in particular the output SDR raised to 17.1 dB for the time-domain
method and 16.5 dB for the penalty-based one, while simply reconstructing thephase as a post-processing does not solve the problem (higher ψ̃, lower SDR).
The increase in SDR may not seem straightforward, but it can be understood asa result of the fact that with our methods the spectrogram of the resynthesized
signal is closer to the intended ML solution. Computation of the analytical timedomain solution is very costly, but enables us to see that the solution obtainedin much less time with the penalty-based algorithm is close to optimal. We will
use this last algorithm for the noise reduction experiments below.
We also studied the inﬂuence of the frame shift on performance, and noticed
that the output SDR increases with the amount of overlap between frames in the
STFT, especially for the analytical solution. This could be expected as consistency constraints become stronger when overlap increases. Computation time ofcourse increases as well, roughly linearly with the total number of spectrogram
frames for all the methods. Detailed results are skipped due to space constraints.
4.3

Real-World Background Noise Reduction

In order to test our method in more realistic conditions, we performed noise
reduction experiments on speech mixed with various types of noise, assumingthat only the average power spectrum of the noise is known and that the powerConsistent Wiener Filtering

95

Table 2. Performance comparison for noise reduction. Values are in dB.
Input SDR
Ga
Su
Sq
Ca

WienerPenalty
WienerPenalty
WienerPenalty
WienerPenalty-10 dBSDR ISR SIR SAR-3.1 12.9 -3.2 6.1
3.4 7.4 11.6 2.5-5.9 18.2 -5.4 6.5-2.8 13.6 -0.7 2.3-4.6 14.6 -4.4 5.7-1.7 9.5 0.1 1.0-4.8 9.7 -5.9 5.6-1.0 4.5 -0.6 -0.9

0 dB
SDR ISR SIR SAR
6.2 20.3 7.3 12.0
9.4 15.6 18.8 10.2
3.9 26.9 4.9 11.5
6.7 23.5 9.8 10.0
5.1 23.6 6.3 11.5
7.1 17.8 11.4 9.2
4.7 18.5 5.4 11.0
6.1 12.2 11.2 7.2+10 dBSDR ISR SIR SAR
14.8 29.0 16.5 20.0
15.9 24.7 23.9 17.1
13.6 34.9 14.8 20.3
15.6 31.9 19.0 19.0
14.6 32.5 16.0 20.4
15.6 27.1 20.3 18.1
13.8 29.1 15.1 19.7
14.0 24.0 19.2 16.0

spectrogram of speech is estimated by spectral subtraction [5]. We considered
four noise signals: a synthetic Gaussian white noise, and three real-world background noises from SiSEC 2010’s “Source separation in the presence of real-world
background noise” task [10] recorded near a subway car (“Su”), on a square(“Sq”) and in a cafeteria (“Ca”). The stereo signals were downmixed to monoand cut to 10 s length to match the speech signals, which were the same as above.
We considered 30 mixtures for each noise, with 10 diﬀerent speech signals and at
three input SDRs: −10 dB, 0 dB and 10 dB. The results for the penalty-basedalgorithm and the classical Wiener ﬁlter, averaged for each noise and input SDR
on the 10 corresponding mixtures, are summarized in Table 2.
We can see that the proposed method leads to a signiﬁcant improvement over
Wiener ﬁltering in terms of output SDR, with, averaged on all noises, further
gains of 4.1 dB, 2.4 dB and 1.1 dB respectively for −10 dB, 0 dB and 10 dB input
SDRs. This can be further analyzed as a strong improvement of the SIR, oﬀset,
to a lesser extent, by a deterioration of the SAR and ISR. Note that this trade-oﬀ
between improvement of SIR and deterioration of SAR and ISR can be tuned
through the penalty weight γ depending on the application, as classical Wiener
ﬁltering indeed corresponds to γ = 0. Perceptually, although there remains somemusical noise, the residual noise present in the Wiener ﬁlter estimates is much
weaker with the proposed method.
We believe that the tendency of our algorithm to further suppress the interference compared with the classical Wiener ﬁlter is related to the distributionof the time-frequency bins whose power has not been canceled through spectral
subtraction. This can be simply understood in the particular case where speech
is replaced by silence. If most bins are set to zero, our algorithm will tend tocancel the remaining ones as well, as a consistent solution with most bins equal
to zero in a given neighborhood is likely to be zero on the whole neighborhood,an eﬀect similar to block-thresholding [11], shown to be one of the most eﬀective
denoising methods to date. This is ﬁrst conﬁrmed by the fact that our algorithm
seems to perform quite well on Gaussian noise, whose power is exponentiallydistributed and for which 63 % of the bins are thus set to zero when subtracting
the mean power. We tested this hypothesis informally by looking at synthetic
noises with various power distribution: the improvements of our algorithm over

96

J. Le Roux et al.

classical Wiener ﬁltering decreased as the proportion of bins above the mean
power increased, although we shall skip the details here for the sake of brevity.
Finally, we note that by comparing the spectral subtraction results, obtainedwith a rather crude estimate for the noise power spectrum, with the oracle ones,
we can expect our method’s performance to depend on the reliability of thepower spectrum estimates.

5

Conclusion

We presented a new framework for Wiener ﬁltering and more generally timefrequency masking which takes into account the consistency of spectrograms
to compute the true optimal solution to the Wiener ﬁltering problem. We presented two methods to ﬁnd optimal or near optimal solutions, investigated theirperformance in comparison with previous works, and showed in particular that
our method combined with spectral subtraction outperforms classical Wiener
ﬁltering. Future works include combining our method with more sophisticatedalgorithms for the estimation of the noise power spectrum, and extension of theframework to the multichannel case.

References
1. Diethorn, E.J.: Subband noise reduction methods for speech enhancement. In:
Huang, Y., Benesty, J. (eds.) Audio Signal Processing for Next-Generation Multimedia Communication Systems, pp. 91–115. Kluwer, Dordrecht (2004)
2. Vincent, E., Jafari, M.G., Abdallah, S.A., Plumbley, M.D., Davies, M.E.: Probabilistic modeling paradigms for audio source separation. In: Machine Audition:
Principles, Algorithms and Systems. IGI Global (to appear)
3. Le Roux, J., Ono, N., Sagayama, S.: Explicit consistency constraints for STFT
spectrograms and their application to phase reconstruction. In: Proc. SAPA, pp.
23–28 (September 2008)
4. Le Roux, J., Kameoka, H., Vincent, E., Ono, N., Kashino, K., Sagayama, S.: Complex NMF under spectrogram consistency constraints. In: Proc. ASJ Autumn Meeting, (2-4-5) (September 2009)
5. Boll, S.: Suppression of acoustic noise in speech using spectral subtraction. IEEE
Trans. ASSP 27, 113–120 (1979)
6. Griﬃn, D.W., Lim, J.S.: Signal estimation from modiﬁed short-time Fourier transform. IEEE Trans. ASSP 32(2), 236–243 (1984)
7. Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Proc.
NIPS*2000, pp. 556–562. The MIT Press, Cambridge (2001)
8. Vincent, E., Gribonval, R., Plumbley, M.D.: BSS Oracle Toolbox Version 2.1,
http://bass-db.gforge.inria.fr/bssoracle/
9. Vincent, E., Sawada, H., Boﬁll, P., Makino, S., Rosca, J.P.: First stereo audio source
separation evaluation campaign: Data, algorithms and results. In: Proc. ICA, pp.
552–559 (September 2007)
10. Araki, S., Ozerov, A., Gowreesunker, V., Sawada, H., Theis, F., Nolte, G., Lutter,D., Duong, N.Q.: The 2010 signal separation evaluation campaign (SiSEC 2010)–Part II–: Audio source separation challenges. In: Proc. LVA/ICA (2010)
11. Yu, G., Mallat, S., Bacry, E.: Audio denoising by time-frequency block thresholding.
IEEE Trans. Signal Process. 56(5), 1830–1839 (2008)