➠➡
REDUCING MUSICAL NOISE BY A FINE-SHIFT OVERLAP-ADD METHODAPPLIED TO SOURCE SEPARATION USING A TIME-FREQUENCY MASKShoko ArakiShoji Makino

Hiroshi SawadaRyo Mukai

NTT Communication Science Laboratories, NTT Corporation
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237, Japan
Email: shoko@cslab.kecl.ntt.co.jp
ABSTRACTMusical noise is a typical problem with blind source separation using a time-frequency mask. In this paper, we reportthat a ﬁne-shift and overlap-add method reduces the musical noise without degrading the separation performance.
The effectiveness was conﬁrmed by results of a the listening test undertaken in a room with a reverberation time of
RT60 =130 ms.
1. INTRODUCTION
In this paper, we consider the blind source separation (BSS)of speech signals realized by utilizing a time-frequency mask.
The time-frequency mask is widely used especially to solve
the underdetermined BSS problem where N source signals
outnumber M sensors. We can extract/separate signals byusing the time-frequency mask; however musical noise causedby the mask is a typical problem.
Several methods have been proposed for underdetermined
BSS (e.g., [1–4]), and they rely on the sparseness of thesource signals. In [1], the authors employ a time-frequency
binary mask to extract each signal from the mixtures. The
use of binary masks causes too much discontinuous zeropadding to the extracted signals, and they contain loud musical noise.
The authors of [2] also deﬁne a time-frequency binary
mask by employing the ratio between an observation andthe output of an adaptive beamformer, which reduces thetarget signal. Then they extract the target signal with thistime-frequency binary mask. In a previous work [3], weused a time-frequency binary mask to remove the N − M
signals from the observations. We then separated the extracted mixtures by independent component analysis (ICA).
Moreover, an ML estimation based ICA method has been
widely studied recently (e.g.,[4–6]), where the sources are
estimated after mixing matrix estimation with the l 1 -norm
minimization approach.
With these approaches [2–6], there is less zero-paddingto the extracted signals than with the binary mask only approach [1]. However, with the N − M source removal
method [3] and with the l 1 -norm approach [4–6], the N −M
components still become zero at each time-frequency point.
Therefore, we still hear musical noise in their outputs.

0-7803-8874-7/05/$20.00 ©2005 IEEE

Musical noise has been widely considered in the ﬁeld ofsingle channel speech enhancement with spectral subtraction (e.g.,[7, 8]). It is said that musical noise is heard when
an output has isolated peaks and/or short ridges in its spectrogram [8]. In underdetermined BSS, we should also consider the musical noise problem because source separation
with a time-frequency mask is also a sort of speech enhancement. In this paper, we report that musical noise is reduced
by a ﬁne-shift and overlap-add method. By using the ﬁneshift and overlap-add method, we attempt to obtain a gradual change of the spectrogram and reduce musical noise.
The effectiveness was conﬁrmed by undertaking a listening
test in a room with a reverberation time of RT 60 =130 ms.
2. UNDERDETERMINED BSS
2.1. Formulation
In real environments, N source signals s i observed by Msensors are modeled as convolutive mixtures
N Lhji (l)si (n − l + 1) (j = 1, · · · , M ),xj (n) =i=1 l=1

where hji is the L-taps impulse response from a source i toa sensor j. The goal of underdetermined BSS (N > M )is to obtain separated signals y k (n) (k = 1, · · · , N ) using
only the information provided by observations x j (n). In
the underdetermined scenario, sources are assumed to havea sparse distribution.
This paper employs a time-frequency domain approach
because speech signals are sparser in the time-frequency domain than in the time domain [5] and convolutive mixture
problems can be converted into instantaneous mixture problems at each frequency. In the time-frequency domain, observed signals are modeled asN
Xj (f, m) =Hji (f )Si (f, m) (j = 1, · · · , M ),i=1

where Hji (f ) is a transfer function from a source i to asensor j, Si (f, m) and Xj (f, m) denote short-time Fouriertransformed sources and observed signals, respectively. f
is the frequency and m is the time-dependence of the shorttime Fourier transformation (STFT).

III - 81ICASSP 2005

➡

➡
101099(a) TR = 0ms

1@1969HzMf(n)occurency

887766(a)0.5(C)(A)

5504200

44334400

4600

1

1120

20

4060

4060~
θ180

100

80

100~
θ2120 140 160 180 θ~ DOA (deg.)
θ3120

140160180

Mf(n)0000Fig. 1. Example histogram. Male-male-female combination withSTFT frame size T = 512. RT60 = 0 ms.

5200

5400

0.5(B)
04200

4400

(D)
4600

Xj (f, m) =T−1
w(r)xj (r + mR)e−j2πf r

(1)

r=0

where f = (0, T1 fs , · · · , T T−1 fs ), fs is a sampling frequency,w(r) is a window and R is the window shift size. Here, we
use a shift of R = T /S where S is the shift rate.
Then a time-frequency mask is estimated using the sparseness of sources. When signals are sufﬁciently sparse, wecan assume that sources do not overlap very often. Therefore, we can classify the observation sample points X j (f, m),and extract each signal with the time-frequency mask. To
classify the observations, we use the estimated direction ofarrival (DOA) θ(f, m) = cos −1 ϕ(f,m)c
2πf d where ϕ(f, m) =Xi (f,m)
Xj (f,m) (i = j) is the phase difference between two ob-servations, d is the microphone space, and c is the speed ofsound. The DOA histogram has N clusters (Fig. 1) and eachcluster corresponds to one source. Using the average DOAsof these clusters θ̃1 , θ̃2 , · · · , θ̃N , we deﬁne a time-frequency
binary mask10θ̃k − ∆ ≤ θ(f, m) ≤ θ̃k + ∆otherwise(2)which extracts the estimated signal of S k (f, m). In theequation, ∆ is an extraction range parameter and here weuse the standard deviation σ k of cluster k for this parameter.
2.3. Output reconstruction with overlap-add method
Next with the time-frequency mask (2), we obtain the output
signal Yk (f, m) = Mk (f, m)Xj (f, m)(j ∈ {1, · · · , M })at each time-frequency point.
Finally, the output signals in the time domain are reconstructed using the overlap-add method,
1  m+lyk (n)
CS−1yk (n) =l=04800

5000

5200

5400

time n (ms.)

Although the ﬁne-shift and overlap-add method should bevalid for various approaches [1–6], here we utilize a basic
binary mask method such as that described in [1].
First, time domain signals xj (n) are converted into thetime-frequency domain with a T -point STFT:

Mk (f, m) =5000(b)2.2. Time-frequency mask estimation

4800

time n (ms.)22(3)

Fig. 2. Examples of overlapped time-frequency mask. (a)S = 2,(b)S = 8. T = 512.

where C is a constant that is decided by the window w(n)and the shift rate S,⎧⎨ w(n − mR)y(n) = f Y (f, m)ej2πf r
ykm (n) =(mR ≤ n ≤ mR + T − 1)⎩0 (otherwise),and r = n − mR. We should utilize an appropriate window
w(n) for the overlap-add. That is, the window should satisfy the condition that the sum of all the windows in y km (n)S−1should be 1: l=0 w(n − (m + l)R) = 1. Here we utilize
the hanning window w(n) = 0.5 − 0.5 ∗ cos( 2πnT ), (n =0, · · · , T − 1) and C = 0.5S, which fulﬁll the condition.
3. PROPOSAL: UTILIZING A FINE-SHIFT
3.1. Fine-Shift
In (1), a half shift R = T /2 (S=2) is usually used. However,
we can also use a smaller shift R = T /S (S > 2). In thispaper, we use a smaller shift, i.e., a ﬁne-shift for the wholeprocess (Secs. 2.2 and 2.3), to see how it affects musicalnoise.
From (3), we can see that the output signal y is obtained
as the sum of S frames. When we use a ﬁne-shift S >2, the output signal is averaged over S frames and can besmoothed. Therefore, we can expect smoothed outputs, i.e.,less musical noise.
3.2. Effect of ﬁne-shift and overlap-add
At each frequency f , the time-frequency mask M (f, m) extracts a signal of duration T (mR ≤ n ≤ mR + T − 1) andthe extracted signals are overlapped according to (3). In order to see what happens in each frequency, we look at anoverlapped time-frequency mask in this section. Figure 2
shows an example of an overlapped time-frequency maskn
R
1
Mf (n) =M (f, m)(4)
Sn−T +1
m= R at a frequency of 1078 Hz.

III - 82

➡

➡
4.45 mTable 1.

Results for RT60
=0 ms. S: shift rate

Loudspeakers(height : 1.35 m)S248S3

1.80 mo

135
2.25 mo

3.55 m

1.1 m

45o

1.75 m

S2904 cm

SIR
17.0
18.3
18.7

SDR
7.9
8.7
9.0

Table 2.

Results for RT60
=130 ms. S: shift rate

MOS
1.9
2.8
3.3

S
2
48

SIR
12.2
13.8
14.3

SDR
6.4
7.4
7.7

MOS
1.6
2.5
3.1

S1

mance and sound quality, respectively:

Omni-directional microphones(height : 1.39 m)

y 2 (n)SIRi = 10 log  (n isyi (n))2 ,isjni=j
x2 (n)
n ksi,
SDRi = 10 log  (x (n)−αy(n−D))2

Room height : 2.50 mFig. 3. Room for reverberant tests. RT60 = 130 ms.

nOne shift length is R = T /S = 32 ms for S = 2 and8 ms for S = 8, because we used T = 512 at a sampling
frequency f s = 8 kHz. When S = 2 we observe gaps of
32 ms [(A) and (C) in Fig. 2]. Our auditory system can recognize a gap of over around 22 ms for the sound at 500 Hz[9]. Therefore, we think that these gaps can be recognizedas musical noise. However, when S = 8, the gap length is
8 ms [(B) in Fig. 2], and so the gap cannot be recognized.
Moreover, at (D) in Fig. 2, the overlapped mask does not
become zero, and its value changes gradually around (D).
We believe this is why we can reduce musical noise withthe ﬁne-shift.
Moreover, the magnitude of one step is 0.125 when S =8, and 0.5 when S = 2 (This is always true because M (f, m)∈ {1, 0}). That is, we rarely have a sudden change of mask
level when S = 8 and therefore we rarely have isolatedpeaks and/or short ridges in the output spectrogram. This isalso one of the reasons for the musical noise being reducedby the ﬁne-shift.
4. EXPERIMENTS
4.1. Experimental conditions
We evaluated the separation performance in anechoic tests
and reverberant tests. For the anechoic tests (RT 60 = 0 ms),
we mixed speech signals using the transfer function H ji (f ) =dexp (j2πf τji ), where τji = cj cos θi , dj is the positionof the j-th microphone, c is the speed of sound, and θ i isthe direction of the i-th source. The source directions were45◦ , 90◦ and 135 ◦. For the reverberant tests, the speech
data were convolved with impulse responses recorded in
the room illustrated in Fig. 3 whose reverberation time was
RT60 = 130 ms. As the original speech, we used Japanese
sentences of around 7 seconds in length spoken by maleand female speakers. We investigated three combinationsof speakers and averaged the results.
The STFT frame size T was 512 at a sampling rate of
8 kHz. We changed the frame shift from 64(= T /8) to256(= T /2) .
4.2. Performance measures
We used the signal to interference ratio (SIR) and the signal
to distortion ratio (SDR) as measures of separation perfor-ksi

isi

where yi is an estimation of si , and yisj is the output of thewhole separating system at y i when only s j is active, andxksi = hki ∗ si (∗ is a convolution operator). α and D areparameters used to compensate for the amplitude and phasedifference between x ksi and yisi .
Since SDR cannot evaluate musical noise [7], we also
conducted a subjective test and obtained the mean opinion
score (MOS). The listening tests were undertaken with 20listeners. Each listener was asked about the audibility ofmusical noise, and they awarded a score from one (clearlyaudible) to ﬁve (not audible) for each output signal.
The SIR and SDR values and the MOSs were averaged
over three speaker combinations.
4.3. Experimental results
Tables 1 and 2 show the SIR and SDR results when wechanged the shift rate S. We can see that the SDR values
increase without any reduction in the SIR values. Note thatthe SIR values are not reduced by the ﬁne-shift.
Moreover, Tables 1 and 2 also show the MOS. An analysis of variance conﬁrmed that there were signiﬁcant differences among the MOS for each shift rate (F (2, 537) =160.8, p < .0001 when RT 60 =0 ms and F (2, 537) = 136.1,p < .0001 when RT 60 =130 ms). Our subjective test conﬁrmed that the ﬁne-shift reduced the audible musical noise.
We also conﬁrmed that the ﬁne-shift is effective for reducing musical noise when with Roman’s method [2] andAraki’s method [3]. Some sound examples can be found at[10].
5. DISCUSSIONS
We can reduce musical noise by using a ﬁne-shift. This isbecause output signals are averaged over S frames. Figure
4 shows spectrograms for S = 2 and S = 8. We can see
the isolated peaks and ridges when S = 2. By contrast, theisolated components are smoothed when S = 8.
Figure 5 shows examples of smoothed signals at a frequency of 1078 Hz. ’org’ and ’shift2’ and ’shift8’ show theamplitudes of the original signal, and the outputs |Y (f, m)|when S = 2 and S = 8, respectively. The plot labeled’LPF(butt)’ is the amplitude of Y (f, m) ﬁltered with a low
pass ﬁlter (Butterworth ﬁlter) when S = 2. Although we

III - 83

➡

➠
0.4

0.4

4000

4000

3500

3500

0.2

0.2

3000

3000

0.1

0.1

2500

2500

0

2000

0.4

1500

0.3

org

frequency (Hz)frequency (Hz)

0.3

2000
1500
1000

1000

500

500

2time (sec)

4

0
0

2
time (sec)

4

1

2

3

4

01

2

3

4

0.4

shift2

0.2

0.1

0.1
01

2

3

smooth2

0.3

4

001

2

3

4

0.4

0.4

smooth4

shift80.3

Fig. 4. Example spectrograms. Left: S = 2, right: S = 8.

00.2

0000

LPF(butt)

0.3

0.2

0.2
0.1
0

expected the smoothing effect with LPF, there was an inappropriate increase in the output signal.
’smooth2’ and ’smooth4’ are the amplitudes of the averaged waveforms of Y (f, m) when S = 2, that is, Y (f, m) ←K−1
1
l=0 Y (f, m−l) and K = 2 for ’smooth2’ and K = 4
Kfor ’smooth4’. The smoothed waveforms have smaller amplitudes than the ’shift2’, and the waveform has changed in’smooth4’. This is because the speech signal is very sparseat each frequency, making this smoothing unsuitable. Moreover, with ’smooth2’ and ’smooth4’, the output Y (f, m) isaveraged at a two-times over-sampling rate of T /S (S = 2).
This is different from the sum of (3) when S = 4 (’shift4’)
or S = 8 (’shift8’), that is, four-times or eight-times oversampling.
We may also be able to use a temporally smoothed timefrequency mask M (f, m), which we can obtain, for example, by shading a binary mask. However, we may pick up
interference at such shaded time-frequency points. Therefore, the SIR values may decrease. By contrast, the overlapadd with the ﬁne-shift method estimates the time-frequency
mask M (f, m) at each (over-sampled) time point m so that
only one signal is picked up. Therefore, the ﬁne-shift does
not affect the SIR values.
Note that the ﬁne-shift increases the calculation cost because the ﬁne-shift is the same as over-sampling. We should
select the shift rate S according to the application.
6. CONCLUSION
We evaluated the effectiveness with which a ﬁne-shift reduces musical noise, which is a problem in the underdetermined BSS. The ﬁne-shift and overlap-add method reduces musical noise effectively without destroying the separation performance. We have already proposed using a spatially continuous mask [11] to reduce musical noise. Theﬁne-shift and overlap-add method realizes a temporally relatively continuous mask. Therefore, we expect that we canreduce musical noise more effectively by using a mask thatis both spatially and temporally continuous.

01234001

23time (s)time (s)Fig. 5. Signal example at a frequency (1078Hz). ’org’:original,’shift2’:S = 2, ’shift8’:S = 8, ’LPF(butt)’:signal ﬁltered by Butterworth ﬁlter, ’smooth2’ and ’smooth4’: average for 2 or 4 sample
points respectively.

7. REFERENCES[1] O. Yilmaz and S. Rickard, “Blind separation of speech
mixtures via time-frequency masking,” IEEE Trans. on SP,
vol. 52, no. 7, pp. 1830-1847, 2004.
[2] N. Roman and D. Wang, “Binaural sound segregation formultisource reverberant environments,” Proc. ICASSP2004,
pp. 373-386, 2004.
[3] S. Araki, S. Makino, A. Blin, R. Mukai and H. Sawada, “Blindseparation of more speech than sensors with less distortion
by combining sparseness and ICA,” Proc. IWAENC2003, pp.
271–274, 2003.
[4] F. J. Theis, C. G. Puntonet and E. W. Lang, “A histogrambased overcomplete ICA algorithm,” Proc. ICA2003,
pp. 1071-1076, 2003.
[5] P. Boﬁll and M. Zibulevsky, “Blind separation of more
sources than mixtures using sparsity of their short-time
Fourier transform,” Proc. ICA2000, pp. 87-92, 2000.
[6] S. Winter, H. Sawada, S. Araki and S. Makino, “Overcomplete BSS for convolutive mixtures based on hierarchical clustering,” ICA2004, (to appear).
[7] N. Virag, “Single channel speech enhancement based onmasking properties of the human auditory system,” IEEETrans. on SAP, vol. 7, no. 2, pp. 126-137, 1999.
[8] Z. Goh, K-C. Tan and B. T. G. Tan, “Postprocessing methodfor suppressing musical noise generated by spectral subtraction,” IEEE Trans. on SAP, vol. 6, no. 3, pp. 287-292, 1998.
[9] B. C. J. Moore, An introduction to the psychology of hearing,
3rd Ed., Academic Press, 1989.
[10] http://www.kecl.ntt.co.jp/icl/signal/araki/ﬁneshift.html[11] S. Araki, S. Makino, H. Sawada and R. Mukai, “Underdetermined blind speech separation with directivity pattern based
continuous mask and ICA,” Proc. EUSIPCO2004, pp. 19911994, 2004.

III - 844