Speech enhancement with ad-hoc microphone arrayusing single source activity
Ryutaro Sakanashi∗ , Nobutaka Ono† , Shigeki Miyabe‡ , Takeshi Yamada§ and Shoji Makino¶∗‡§¶ Graduate School of Systems and Information Engineering, University of Tsukuba, Japan† National Institute of Informatics / School of Multidisciplinary Sciences,

The Graduate University for Advanced Studies (SOKENDAI) , Japan
E-mail: ∗ sakanashi@mmlab.cs.tsukuba.ac.jp, † onono@nii.ac.jp,{‡ miyabe, ¶ maki}@tara.tsukuba.ac.jp, § takeshi@cs.tsukuba.ac.jp
Abstract—In this paper, we propose a method for synchronizing asynchronous channels in an ad-hoc microphone arraybased on single source activity for speech enhancement. An adhoc microphone array can include multiple recording devices,which do not communicate with each other. Therefore, theirsynchronization is a significant issue when using the conventional
microphone array technique. We here assume that we know two
or more segments (typically the beginning and the end of therecording) where only the sound source is active. Based on thissituation, we compensate for the difference between the start andend of the recording and the sampling frequency mismatch. We
also describe experimental results for speech enhancement witha maximum SNR beamformer.

unknown gains, and each device starts recording independently. In particular, the sampling frequencies are not common
to all the observation channels because of independent A/Dconverters, and sampling frequency mismatches are inevitable.
The difference between the unit lengths of samples causes
the time difference between observed digital signals in different channels to drift. Since most of array signal processing
methods assume that the locations of sound sources haveunique time differences of arrival (TDOAs) among observation
channels, even a sample of change in the TDOAs is very large
for array signal processing.

I. I NTRODUCTION
A microphone array using multiple microphones can perform various types of signal processing by obtaining spatial
information from the phase difference of the sound waves
that reach the microphones. There has been rapid progressin research on the application of this technology to handsfree speech recognition and the understanding of the soundenvironment by computers in real environments. For example,
a beamformer enhances target speech by controlling directional characteristics. Also, blind source separation extracts
speech sources from the mixed observation of multiple sources
without prior information. Generally, these microphone arraysignal processing techniques assume that the microphoneelements are placed regularly and the recording channels aresynchronized properly with unified multichannel A/D converters, and such requirements result in the limited applicabilityof microphone arrays because of the need to use special
expensive equipment.
To extend the application of microphone array signal processing, increasing attention has been paid to ad-hoc microphone arrays, which use multiple independent recording
devices for multichannel speech signal processing. The advantage of the ad-hoc microphone array is that it gives us
freedom when choosing recording devices for many-channel
recording, and it requires no large-scale recording devices
such as special microphones or many-channel analog-to-digitalconverters (ADCs). However, asynchronous channels have
many additional issues that are not dealt with in conventional
microphone array signal processing. For example, the arraygeometry is unknown, the recording devices have different

Several studies have tried to deal with this problem. On
the assumption that there was no sampling frequency mismatch, some authors proposed blind alignment to estimate
the recording start time and the positions of microphones andsources simultaneously. Robledo et al. examined compensationof the sampling mismatch by resampling with interpolation.
For blind estimation of sampling mismatch, Liu et al. utilized
the correlation of an amplitude spectrogram. Markovich et al.
proposed the semi-blind compensation of sampling mismatchwith given speech absence information. Recently, we proposed
the accurate blind compensation of sampling mismatch assuming stationarity of observation.
In this paper, we propose a user-guided speech enhancement
framework in an ad-hoc microphone array scenario assumingthat two short intervals of target speech activity are specified
by the user. By estimating the inter-channel time difference ofthe specified intervals, the identification of the sampling mismatch is no longer a blind estimation problem. The intervals
are also used for the adaptation of a maximum signal-to-noise
ratio (SNR) beamformer for speech enhancement. Although
the well-used approach to adaptation with a steering vector is
not easy in the distributed microphone array scenario where
the positions of speakers and microphones are unknown, amaximum SNR beamformer optimizes its directivity using
speech activity information without the steering vector. Experimental results show that our proposed method enhances
the target speech successfully by employing the multichannel
attribute of an ad-hoc microphone array.

II. T IME DOMAIN MODEL OF ASYNCHRONOUS RECORDING
First, we discuss the formulation of the drift in asynchronous recording. Although we limit the discussion to thesampling frequency mismatch between two channels in thispaper, it is easy to extend it to three or more channels byfitting sampling frequency of all channels to one specificchannel. Suppose that sound pressures x1 (t) and x2 (t) on
two microphones are sampled by different ADCs as x1 (n1 )and x2 (n2 ), where t denotes continuous time and n gives
the discrete time. Also suppose that the sampling frequencyof x1 (n1 ) is fs , and that of x2 (n2 ) is (1 + ϵ)fs with adimensionless number ϵ. This paper assumes that the ADCs
have common nominal sampling frequencies and |ϵ| ≪ 1.
Then the relations between xi (n) and xi (t) for i = 1, 2 aregiven by( )n1
x1 (n1 ) = x1
(1)
f
)
( s
n2
x2 (n2 ) = x2
+ ∆T21(2)(1 + ϵ) fs
Where ∆T21 is the time at which the sampling of x2 (n2 )starts. Here, the sample number that refers to the same time tof channel i (i = 1, 2) is given by
n1 = tfs ,n2 = (1 + ϵ)(t − ∆T21 )fs .

(3)
(4)

Then, n2 is expressed with n1 as below,
n2 = (1 + ϵ)n1 − (1 + ϵ)D21 ,(5)

where D21 = ∆T21 fs stands for the discrete time of the firstchannel when the recording of the second channel starts. And
the difference between n1 and n2 is given by
ϕ(n1 ) = n2 − n1 = ϵn1 − (1 + ϵ)D21 .

(6)

The difference ϕ(n1 ) in the number of samples between
two signals has a proportionate relationship, and increases
with time. This causes the source image to drift and isequivalent to the source position moving artificially. Suchmovement disrupts conventional microphone array techniques
which utilize the time difference of arrival (TDOA) explicitly
or implicitly to control directivity. Therefore, it is necessary
for the ad-hoc microphone array to estimate the sampling
frequency mismatch ϵ.
III. S UPERVISED IDENTIFICATION AND COMPENSATION OF
SAMPLING MISMATCH

In our proposed framework in which short intervals where
only a target source is active are given for the speech enhancement, the estimation of the sampling mismatch is no longer anunsupervised estimation problem. This section describes our
proposed supervised identification of the sampling mismatch.

Ch1

Sampling
frequency:(reference
signal)Ch2

Sampling
frequency:(compensation signal)

Fig. 1. Mismatch model.

A. Identification of sampling mismatch using single-sourceactive short intervals
Before proceeding to a discussion of the estimation procedure, we show that both the sampling frequency mismatch ϵand the recording start offset D21 are identifiable if two pairs{nA1 , nA2 } and {nB1 , nB2 } of times corresponding to the same
analogue times are available. These variables have to satisfy
the following conditions.
nA2 = (1 + ϵ) (nA1 − D21 ) ,nB2 = (1 + ϵ) (nB1 − D21 ) ,The conditions identify ϵ and D21 as
nB2 − nA2ϵ=
− 1,
nB1 − nA1
nA1 nB2 − nA2 nB1
D21 =.
nB2 − nA2

(7)
(8)

(9)
(10)

Thus by estimating two pairs of corresponding times nAi andnBi , i = 1, 2, we can obtain the estimate of ϵ and D21 .
Since precise estimation of these time pairs is difficult andthe estimation necessarily has errors, it is preferable that thenA1 , nA2 values are small and those of nB1 , nB2 large.
Now we discuss how to identify a sampling mismatch
from a single source activity. We assume that we have two
short intervals when one of the sources is active near the
beginning and one is active of the end of the recording foreach channel, and the estimation is accomplished by analyzing
the time difference to maximize the correlation between thechannels, and estimate the synchronous time pairs includedin the specified intervals, as shown in Fig. 1. However, there
are two issues which mean that the estimation cannot be anexact one. The first issue is that the correlation gives only
the TDOA of each interval, which affects both the samplingmismatch and the relative positioning of the microphones andthe source. Thus it is hard to evaluate only the effect of
the sampling mismatch. However, the TDOA caused by the
positioning is constant when the source does not move, andwe ignore its effect. Although the estimation of the sampling
frequency mismatch ϵ is not effected but the recording start
offset D21 is given a small error. This error is problematicwhen the direction of arrival (DOA) is explicitly used inarray signal processing. However, it is not problematic in our
scenario because the maximum SNR beamformer that we use
in the speech enhancement stage uses only the source activity
information and the DOA information is not required. The
second issue is that the specification of only the intervals is
not sufficient for the specification of the synchronous times
because where exactly in the single source is located in theroughly specified interval by user’s hands. Hereafter, we call

Rough cuts
Single source activity

(a)

(b)

Ch1
(reference
signal)Ch2(compensated signal)
Fig. 3. Correlation in rough cuts.
Fig. 2. Two pairs of rough cuts.

and can be approximated under the condition |mϵ| ≪ 1 as
the roughly specified interval the rough cut. We discuss thisissue and show how to minimize the n1 and n2 estimation
error in the rough cuts in the following.
Suppose we have a pair of rough cuts of the two channels,
and the length of both rough cuts is I. The rough cuts aredenoted by i1 , . . . , i1 + I − 1 for the first channel and i2 +2, . . . , i2 + 2 + I − 1 for the second channel. As shown in Fig.
2, the time difference of the speech activity in the rough cuts
is estimated as
δ21 = arg max
τ

I−1
∑

x1 (i1 + l) x2 (i2 + l − τ ) .

(11)

l=0

By ignoring the TDOA caused by the positions as discussed
above, the following relation can be assumed.
n2 − n1 = δ21 .

(12)

However, the size of n1 − i1 remains unknown, as shownin Fig. 3 (a). Therefore, as a safe choice of the estimation,
we assume that the speech activity is located in the center inaverage as
n1 + n2 = i1 + i2 ,(13)and obtain the estimate as1n1 = i1 + I/2 + δ21 ,(14)2
1n2 = i2 + I/2 − δ21 ,(15)
2
as shown in Fig. 3 (b). Although the error remains in theestimate, its effect is reduced by making nA1 and nA2 small,and nB1 and nB2 large.

ϕ21 (n + m; ϵ, D21 ) ≈ ϕ21 (n; ϵ, D21 ) + m.

(17)

Thus the discrete times n1 + m and n2 + m of the twochannels near the synchronous pair n1 and n2 can be regarded
as synchronous.
Therefore, a frame analysis xfri (l, ni ) , l = 0, . . . , L − 1 of
the i-th channel of length L (throughout this paper we assume
L is even) centered at ni , given by)(Lfr
xi (l, ni ) = w (l) xi l + ni −,(18)2where w (l) is an appropriate window function, is almost synchronous between the channels i = 1, 2. Since the sampling
frequency mismatch ϵ is generally in the order of 10−5 and thetypical frame length for microphone array signal processing
is in the order of 0.1 second, the largest approximation error|ϵL| /2 of the time in such a frame analysis is usually in theorder of 1 µs. Since such the worst error appears near the
beginning and the end of the frame, the influence of the errors
is reduced by choosing a typical window function w (l) to
suppress the amplitude near both ends.
C. Synchronization by frame analysis of non-integer sample
shift
Here we discuss the STFT expression of the approximationof xfr2 (l, n2 ) assuming ϵ and D21 are given. The STFT analysisof the i-th channel of the frame centered at the sample n isgiven by
(
)
L−1
∑
2πȷkl
fr
Xi (k, n) =xi (l, n) exp −
,
(19)
L
l=0

B. Modeling sampling frequency mismatch in short-timeframes
Before we proceed to the STFT analysis, we discuss theeffect of drift in a short-time frame. We show that the sampling
frequency mismatch can be disregarded in a short interval.
The discrete time of the second channel synchronous withthe (n1 + m)-th sample of the first channel is given by the
relation in (6) as
ϕ21 (n1 + m; ϵ, D21 ) = (1 + ϵ) (n1 − D21 ) + (1 + ϵ) m
= ϕ21 (n1 ; ϵ, D21 ) + (1 + ϵ) m, (16)where k = −L/2, . . . , L/2 − 1 is the discrete frequency
index. Note that the transform is calculated by using a fast
Fourier transform in practical processing. According to (6), thediscrete time of the second channel synchronous to the central
time n1 of X1 (k, n1 ) is given by n2 = ϕ21 (n1 ; ϵ, D21 ). In[6], we approximated the STFT of the second channel centeredat the non-integer time with the following equation.
X̃2 (k, ϕ21 (n1 ; ϵ, D21 )) =)(2πȷk (ϕ21 (n1 ; ϵ, D21 ) − n1 ).
X2 (k, n1 ) exp
L(20)

However, this linear phase compensation assumes that thesize |ϕ21 (n1 ; ϵ, D21 ) − n1 | of the shift is much smaller than
the frame size L, which cannot be maintained with longobservation. To avoid the error that arises with the mismatch of
the assumption, we apply the frame analysis with the nearest
integer central time, and compensate for the effect of therounding by the circular time shift using a linear phase filter.
The integer sample ϕ̄21 (n1 ; ϵ, D21 ) nearest to the desired
central time ϕ21 (n1 ; ϵ, D21 ) is given byϕ̄21 (n1 ; ϵ, D21 ) = arg min |ϕ21 (n1 ; ϵ, D21 ) − n| .

(21)

n∈Z

Since the( central sample ϕ̄)21 (n1 ; ϵ, D21 ) of the short-timeframe xfr2 l, ϕ̄21 (n1 ; ϵ, D21 ) is delayed from the non-integer
time ϕ21 (n1 ; ϵ, D21 ) by ϕ̃21 (n1 ; ϵ, D21 ), given byϕ̃21 (n1 ; ϵ, D21 ) = ϕ21 (n1 ; ϵ, D21 ) − ϕ̄21 (n1 ; ϵ, D21 ) , (22)
we obtain the approximation of synchronization in the STFT
domain by compensating for the delay with the linear phasefilter asX̂2 (k, ϕ21 (n1 ; ϵ, D21 )) =()
X2 k, ϕ̄21 (n1 ; ϵ, D21 ) exp(

2πȷk ϕ̃21 (n1 ; ϵ, D21 )L).

To obtain the STFT analysis for array signal processing, thecentral samples n1 of the first channel should be defined with aregular frame shift, and the second channel has to be adjusted.
First, we analyze the first channel as X1 (k, n1 ), n1 = rR,r = 0, 1, 2, . . ., where R is the frame shift appropriate for thesignal reconstruction determined by overlap-and-add analysis,
and r is the frame index. Second, we obtain the STFT analysisof the second channel as X̂2 (k, ϕ21 (n1 ; ϵ, D21 )) with (23).
This STFT of the second channel corresponds to a frame
analysis with a non-integer frame shift (1 + ϵ) R. Note thatsynchronized observed signals can be obtained by an inverse
STFT analysis with the frame shift R.
IV. S PEECH ENHANCEMENT OF ASYNCHRONOUS
RECORDING USING MAXIMUM SNR BEAMFORMER

Next, we describe a maximum SNR beamformer for speechenhancement that employs single source activity. With thesynchronization described in the previous section, it is possible
to control directivity even for asynchronous multichannel
recording. And we are able to achieve optimal speech enhancement that maximizes the SNR. In this section, we adapt the
maximum SNR beamformer using the single source activity
to time corrected signals.
Here, the power ratio λ (ω) is expressed asw(ω)RT (ω)wH (ω).
w(ω)RI (ω)wH (ω)

t∈ΘI

Here, ΘT and ΘI are sets of the time frames of the target
signal interval and the non-target interval, respectively. The
filter w(ω) used to maximize the power ratio λ(ω) is given
as an eigenvector corresponding to the maximum eigenvalueof the following generalized eigenvalue problem;w(ω)RT (ω) = λ(ω)w(ω)RI (ω).

(27)

Since the maximum SNR beamformer w(ω) has a scaling
ambiguity, we revise the beamformer as:w(ω) ← bk (ω)w (ω) ,(28)

where bk (ω) is the k-th component of b(ω) given byw(ω)Rx (ω),w(ω)Rx (ω)wH (ω)
T
1∑
Rx (ω) =x(ω, t)xH (ω, t).
T t=1b(ω) =(29)(30)

Then enhanced signal y(ω, t) is obtained as(23)

λ(ω) =

where, RT and RI are the covariance matrices of the activityof the target signal, which are expressed as
1 ∑RT (ω) =xT (ω, t)xH
(25)
T (ω, t).
|ΘT |
t∈ΘT
1 ∑
RI (ω) =xI (ω, t)xH
(26)
I (ω, t).
|ΘI |

(24)

y(ω, t) = wH (ω)x(ω, t).

(31)

V. E XPERIMENTS
A. Experimental conditions
We evaluate our proposed speech enhancement strategy fora distributed microphone array scenario using real portable
recording devices.
The task is to enhance the desired speech in the mixture
consisting of target and interfering speakers’ voices observedwith two stereo recording devices. The voices are played backfrom different loudspeakers. Since the objective evaluation of
ad-hoc microphone array recording is not simple, we recorded
the speech in a special manner that we describe later. Since theeffect of the drift in the asynchronous recording is considerable
even if the sampling frequency mismatch is small, we observed
a 30-minite-long signal. By recording the same signals for the
training and the evaluation at both the beginning and the endof the 30-minute recording, and we were able to evaluate thefollowing two conditions:a) Adopt the beamformer at the beginning and apply it to
the speech enhancement at the beginning.
b) Adopt the beamformer at the beginning and apply it to
the speech enhancement at the end.
Needless to say, condition b) is the hardest. We compare thefollowing three methods.
i) Beamforming two channels of one device. (Fig. 4)ii) Beamforming signals, where their recording start isroughly aligned using the recordings of the training
intervals. (Fig. 5)

Signal fortraining (5s)

Chirp signal

Ch1 (L)

Sampling
frequency:

Signal forevaluation (30s)First session

Ch1 (R)

Fig. 4. i) Beamforming of two channels of one device.
Ch1 (Stereo)

Second session

Sampling
frequency:(reference
signal)Ch2 (Stereo)

Same signal for training and evaluation

Sampling
frequency:(compensated signal)

Fig. 5. ii) Beamforming the signals with their recording start is roughly aligned
using the recordings of the training intervals.
Ch1 (Stereo)

Sampling
frequency:(reference
signal)Ch2 (Stereo)

Sampling
frequency:(compensated signal)

Fig. 6. iii) Beamforming the signals of the compensated sampling (proposed
method).

Fig. 8. Recorded signal of each session.

[dB]
35.00
30.00
25.00
20.00

i

15.00

ii

10.00

iii

5.00
0.00
SDR

15°

-30°

SIR
a

SANYO
ICR-PS603RM
TASCAM
DR-05

1.6m

SDR

SIR
b

10cm

reverb time: 800ms

Fig. 7. Recording room.

iii) Beamforming the signals of the compensated sampling(proposed method). (Fig. 6)
We show the recording room layout in Fig.7. The reverberation time T60 is 800 ms. We played back female and malespeakers’ voices from two loudspeakers, and we regarded thefemale boice as the desired speech. The recording devices
we used were SANYO ICR-PS603RM and TASCAM DR05. Both of the devices have the nominal sampling frequencyof 16,000 Hz and the quantization resolution of 16 bit. The
lengths of the signals used for training the desired signal andthe interference were 5 ms, and the mixed signal for the evaluation was 30 ms. The frame length and the frame shift wereconsisted of 16,384 samples and 8,192 samples, respectively.
The evaluation scores were the signal-to-distortion ratio (SDR)
as a quality measure and the signal-to-interference ratio (SIR)
as the interference reduction score.
To obtain an ideally separated signal as the reference forthe objective evaluation, we observed each of the sources
separately with the same layout of the microphones andthe loudspeakers, and we synthesized the observed mixture

Fig. 9. Experimental results.

by summing those two observations. However, we have toconsider the different asynchronous conditions of these two
observation sessions; the two asynchronous devices start
recording at different times, and the same difference cannot
be reproduced again. Thus we align the recording start timeof the second observation with that of the first by playing achirp signal at the beginning of both sessions. By giving ashift to the observation of the second session to maximize
the correlation of the two observed chirps, the starts of therecording of the two sessions are aligned. Note that we assume
that the sampling frequency of a device remains unchanged
throughout the two recording sessions. We show the structureof the signals played back in each session. (Fig. 8)
B. Discussion
The experimental results are shown in Fig. 9. According to
the proposed estimation, the sampling frequency mismatch isabout 104 ppm.
The graph shows that recording start time offset compensation alone given as the method ii is insufficient for timesynchronization. Thus it can be said that the effect of thedrift is severe enough to degrade the array signal processingin this situation, and we must compensate for the sampling
frequency mismatch. Needless to say, method i using thetwo synchronized channels is not affected by the drift, and it
performs the speech enhancement successfully throughout therecording. The proposed method given as method iii performs
better than method i. Therefore we can conclude that the
proposed method successfully compensates for the samplingmismatch and utilizes the asynchronous channels effectively
for speech enhancement.
VI. C ONCLUSION
In this paper, we proposed a speech enhancement framework
based on an ad-hoc microphone array using single source
activity. The single source activity is utilized both in the synchronization stage and the subsequent array signal processing
stage. Experimental results showed that the proposed method
effectively uses the asynchronous recording channels with drift
for speech enhancement.
ACKNOWLEDGMENT
This project received the support of the National Instituteof Informatics (NII) as part of their promotion of strategic
research named ”Grand Challenge”.
R EFERENCES[1] O. L. Frost. “An algorithm for linearly constrained adaptive arrayprocessing.” Proc. IEEE, Vol. 60, No. 8, pp. 926–935, August 1972.
[2] S. Makino, T.-W. Lee, and H. Sawada, Eds., Blind Speech Separation,
Springer, 2007.
[3] K. Hasegawa, N. Ono, S. Miyabe, and S. Sagayama, “Blind estimationof locations and time offsets for distributed recording devices,” Proc.
LVA/ICA, pp. 57-64, Sep. 2010.
[4] Enrique Robledo-Arnuncio, Ted S. Wada, and Biing-Hwang Juang, “Ondealing with sampling rate mismatches in blind source separation andacoustic echo cancellation,” Proc. WASPAA, pp.21-24, 2007
[5] Z. Liu, “Sound source separation with distributed microphone arrays in
the presence of clock synchronization errors,” Proc. IWAENC, 2008[6] S. Miyabe, N. Ono, and S. Makino, “Blind compensation of inter-channel
sampling frequency mismatch with maximum likelihood estimation in
STFT domain,” Proc. ICASSP, pp..674-678, 2013.
[7] N. Ono, H. Kohno, N. Ito, and S. Sagayama, “Blind Alignment ofAsynchronously Recorded Signals for Distributed Microphone Array,”
Proc. WASPAA, pp.161-164, Oct., 2009.
[8] S. Markovich-Golan, S. Gannot, and I. Cohen, “Blind sampling rate offset
estimation and compensation in wireless acoustic sensor networks withapplication to beamforming,” Proc. IWAENC, 2012.
[9] H. L. Van Trees, ed., Optimum Array Processing, Wiley, 2002.
[10] S. Araki, H. Sawada and S. Makino, “Blind speech separation in a
meeting situation with maximum SNR beamformers,” Proc. ICASSP, vol.
1, pp.41–44, 2007.
[11] E. Vincent, H. Sawada, P. Boll, S. Makino, and J. P. Rosca, “First
stereo audio source separation evaluation campaign: data, algorithms andresults,” Proc. ICA, pp. 552–559, 2007.