2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

October 20-23, 2013, New Paltz, NY

OPTIMIZING FRAME ANALYSIS WITH NON-INTEGRER SHIFTFOR SAMPLING MISMATCH COMPENSATION OF LONG RECORDING
Shigeki Miyabe1 Nobutaka Ono2 Shoji Makino1
1

University of Tsukuba, Life Science Center of Tsukuba Advanced Research Alliance1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8573, Japan, {miyabe, maki}@tara.tsukuba.ac.jp
2
National Institute of Informatics, Principles of Informatics Research Division
2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, 101-8430, Japan, onono@nii.ac.jp
ABSTRACTThis paper proposes a blind synchronization of ad-hoc microphonearray in the short-time Fourier transform (STFT) domain with the
optimized frame analysis centered at non-integer discrete time. We
show that the drift caused by sampling frequency mismatch of asynchronous observation channels can be disregarded in a short interval. Utilizing this property, the sampling frequency mismatch andthe recording start offset are estimated roughly by finding two pairsof the short intervals corresponding to the same continuous time.
Using the estimate, STFT analysis is synchronized roughly between
channels with optimized frame central. Since the optimized frame
central is generally non-integer, we approximate the frame analysis by the linear phase filtering of the frame centered at the nearest
integer sample. Maximum likelihood estimation refines the compensation of sampling frequency mismatch.
Index Terms— Ad-hoc microphone array, sampling frequency,
maximum likelihood estimation, blind source separation
1. INTRODUCTION
Ad-hoc microphone array has attracted increasing attention in association with development of mobile recording devices [1]. By
using simultaneous recording with separate mobile devices such ascell phones, voice recorders, video camera etc. as multichannel
observation, microphone array technology obtains accelerated applicability. However, increased freedom brings various difficulty.
Especially the asynchronous sampling of each channel causes driftof time, which seriously degrades time-difference analysis of arraysignal processing [2].
While many of the compensation methods of sampling frequency mismatch requires some prior information [1, 2, 3], we proposed fully blind compensation [4]. By disregarding small drift inside short-time frames, the compensation reduced to the small shiftof the frames. To process very small time shift accurately, the compensation was conducted as linear phase filtering in the short-time
Fourier transform (STFT) domain. Also the sampling frequency
mismatch is estimated by maximum likelihood estimation. However, with long recordings the required shift becomes significantly
large inside each frame, and the circular shift inside the frame fails
to approximate the compensation. In this paper we propose an optimization method of frame analysis with non-integer frame shift.
With correlation analysis, short interval pairs indicating the same
continuous times in each channel is detected. By using two pairsof synchronous intervals, the synchronization is estimated roughly.

978-1-4799-0972-8/13/$31.00 ©2013IEEE

The small error of the synchronized STFT analysis is compensatedeffectively by the maximum-likelihood linear phase.
2. PROBLEM STATEMENT
Suppose sound pressures x1 (t) and x2 (t) on two microphones are
sampled by different ADCs as x1 (n) and x2 (n), where t denotes
the continuous time and n gives the discrete time. The sampling
frequency of x1 (n) is fs , and that of x2 (n) is (1 + ϵ) fs with a dimensionless number ϵ. This paper assumes that the ADCs have thecommon nominal sampling frequencies and |ϵ| ≪ 1. The relations
between xi (n) and xi (t) for i = 1, 2 are given by( )n, n = 0, . . . , N1 − 1,(1)
x1 (n) = x1
fs()
n
x2 (n) = x2
+ T21 , n = 0, . . . , N2 − 1, (2)(1 + ϵ) fs
where Ni is the length of the digital signal xi (n). The relation between the synchronous pair of the discrete times n1 , n2 of these two
sampled signals x1 (n1 ), x2 (n2 ) correspond to the same continuous time t is given by a function ϕ21 (n) as
n2 = ϕ21 (n1 ) ,ϕ21 (n) = (1 + ϵ) (n − D21 ) ,D21 = fs T21 ,

(3)
(4)
(5)

where D21 stands for the discrete time of the first channel when
the recording of the second channel starts. Note that hereafter we
use the notation n1 and n2 to denote the pair of the discrete timecorresponding to the same time, and simply use the notation n when
we don’t need to consider such the correspondence. With the integer
values of n1 , the corresponding discrete times n2 of the second
channel are generally non-integer. Thus to obtain the signal x̂2 (n)of the second channel synchronized accurately to the first channel,
the following infinite convolution of the sinc functions is required.
x̂2 (n) = x2 (ϕ21 (n))
∞
∑
(
) ( )
=
sinc (1 + ϵ) (n − fs T21 ) − n′ x2 n′ . (6)n′ =−∞

Since such the infinite convolution cannot be operated in practice,some efficient approximation is necessary.
In this paper we discuss approximate synchronization of theasynchronous channels with blind estimation of D21 and ϵ with accuracy sufficient for array signal processing. Estimation accuracy

2013 IEEE Workshop on Applications of Signal Processing to Audio and Acousticsof ϵ is critical for array signal processing because the drift makes
the time difference of arrival of each source time-varying. In contrast to the sampling frequency mismatch ϵ, estimation accuracy of
D21 is not significant in specific classes of array signal processing
such as blind source separation (BSS) [5] which do not use directions of arrival explicitly, and the error is accepted as long as it ismuch smaller than the frame length.
3. APPROXIMATE COMPENSATION OF SAMPLINGFREQUENCY MISMATCH IN STFT DOMAIN

October 20-23, 2013, New Paltz, NY

where k = −L/2, . . . , L/2 − 1 is the discrete frequency index.
Note that the transform is calculated by fast Fourier transform in
the practical processing. According to (3), the discrete time of thesecond channel synchronous to the central time n1 of X1 (k, n1 )is given by n2 = ϕ21 (n1 ). To approximate the STFT centeredat the non-integer time ϕ21 (n1 ), we apply the frame analysis with
the nearest integer central time, and compensate the effect of therounding by the circular time shift using the linear phase filter.
First we obtain the frame analysis xfr2 (l, ⌊ϕ21 (n1 )⌉) of the second channel x2 (n) centered at the integer sample ⌊ϕ21 (n1 )⌉ nearest to the desired central time ϕ21 (n1 ), given by

Since array signal processing is generally conducted in the STFT
domain, we propose the STFT expression of the approximate synchronization between the channels.

⌊ϕ21 (n1 )⌉ = arg min |ϕ21 (n1 ) − n| , n ∈ Z.

3.1. Modeling sampling frequency mismatch in short-timeframesSince the central sample ⌊ϕ21 (n1 )⌉ is delayed from the non-integer
time ϕ21 (n1 ) by ϕ21 (n1 ) − ⌊ϕ21 (n1 )⌉, we obtain the approximation of synchronization in the STFT domain by compensating thedelay with the linear phase filter as

Before we proceed to STFT analysis, we discuss the effect of thedrift in a short-time frame. We show that the sampling frequency
mismatch can be disregarded in a short interval.
The discrete time of the second channel synchronous to the(n1 + m)-th sample of the first channel is given by the relation in(3) asϕ21 (n1 + m) = (1 + ϵ) (n1 − D21 ) + (1 + ϵ) m
= ϕ21 (n1 ) + (1 + ϵ) m,(7)

and can be approximated under the condition |mϵ| ≪ 1 as
ϕ21 (n + m) ≈ ϕ21 (n) + m.

X̂2 (k, ϕ21 (n1 )) =(X2 (k, ⌊ϕ21 (n1 )⌉) exp

2πȷk (ϕ21 (n1 ) − ⌊ϕ21 (n1 )⌉)
L

)
.
(12)

It is worth noting that the time domain signal x̂2 (n; ϵ, D21 ) approximately synchronized to x1 (n) can be given by the inverse
STFT analysis of X̂2 (k, ϕ21 (n1 )) with the frame shift common
to X1 (k, n1 ).

(8)Thus the discrete times n1 +m and n2 +m of the two channels near
the synchronous pair n1 and n2 can be regarded to be synchronous.
Therefore, a frame analysis xfri (l, ni ) , l = 0, . . . , L − 1 of thei-th channel of the length L (throughout this paper we assume L iseven) centered at ni , given by()
L
xfri (l, ni ) = w (l) xi l + ni −,(9)2where w (l) is an appropriate window function, is almost synchronous between the channels i = 1, 2. Since the sampling frequency mismatch ϵ is generally in the order of 10−5 and typicalframe length of microphone array signal processing is in the order of 0.1 second, the largest approximation error |ϵL/2| of thetime, which appears in the beginning and the end of the frame withm ≈ ±L/2 in (7) and (8), is usually below the order of 1 µs in sucha frame analysis. Note that the influence of the errors are reducedwith typical choice of the window function w (l) to suppress theamplitude near both ends.
3.2. Approximate synchronization in STFT domain
Here we discuss the STFT expression of the approximation of
xfr2 (l, n2 ) by the linear phase filtering of the frame centered at therounded integer sample. Although the linear phase gives the circular convolution inside the frame, its error can be disregarded withthe large frame length L ≫ 1.
The STFT analysis of the i-th channel of the frame centered at
the sample n is given by
)
(
L−1
∑ fr
2πȷkl
,
(10)
Xi (k, n) =xi (l, n) exp −
L
l=0

(11)

n

4. COARSE BLIND SYNCHRONIZATION
In [4] we proposed a blind estimation of sampling frequency mismatch ϵ and its compensation by the linear phase filtering of the
STFT analyzed analogously to the other channel. However, withthe long observation, the linear phase compensation is insufficientbecause the large gaps of the frame centrals. To solve this problem, we propose a roughly synchronized STFT analysis by estimating the recording start offset D21 and the sampling frequency mismatch ϵ roughly and substituting them in the STFT synchronization
discussed in the previous section.
4.1. Estimation with two pairs of corresponding time
Before the discussion of estimation procedure, we analyze the condition that the parameters ϵ and D21 have to satisfy when two pairs{nA1 , nA2 } and {nB1 , nB2 } of synchronous times are given:nA2 = (1 + ϵ) (nA1 − D21 ) ,nB2 = (1 + ϵ) (nB1 − D21 ) ,

(13)
(14)

which give ϵ and D21 as
nB2 − nA2
− 1,
nB1 − nA1
nA1 nB2 − nA2 nB1
D21 =.
nB2 − nA2
ϵ=

(15)
(16)

Thus by estimating two pairs of corresponding times nAi and nBi ,i = 1, 2, we can obtain the estimate of ϵ and D21 . Although accurate estimation of the synchronized time pairs is difficult, even theirrough estimation is useful for the coarse synchronization.

2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

October 20-23, 2013, New Paltz, NY

4.2. Estimating synchronous times with correlation analysis
We discuss the rough estimation of nAi , nBi for i = 1, 2. The basic
idea is that the nearly synchronous pairs of short intervals can bedetected using correlation. Since the observation is the mixture ofmultiple sources with different time differences of arrival, the estimation of intervals has error and precise one cannot be obtained.
Thus to reduce the effect of the error in the parameter estimation in(15) and (16), it is preferable to estimate nAi of small values andnBi of large values. To obtain the estimation of the intervals, we analyze the intervals with the amplitude exceeds the threshold near the
beginning and the end of the recording. Since it is unknown whichchannel starts and ends earlier or later, the candidates of the shortintervals are chosen from both channels near each of the beginningand the end, and choose the better one with the higher correlation
to other channel.
First, as the candidates of nAi , i = 1, 2, we find the earliest
samples n′Aij , (i, j) = (1, 2), (2, 1) with the amplitude exceeding
the threshold δ in the region over N/2 samples later than the beginning sample asn′Aij = arg min {|xi (n)| > δ} + N/2 − 1, n ≥ N/2,

(17)

n

where N shows the length of the short interval. Similarly, as thecandidates of nBi , i = 1, 2, we find the latest samples n′Bij , (i, j) =(1, 2), (2, 1) with the amplitude exceeding the threshold δ in theregion over N/2 − 1 samples earlier than the end of the channel’srecording asn′Bij = arg max {|xi (n)| > δ} − N/2, n < Ni − N/2. (18)n

Next, for each of h = A, B, (i, j) = (1, 2) , (2, 1), we find thesample n′′hij of xj (n) to maximize the correlation with the selectedinterval as
∑

N/2−1

n′′ij = arg max
n′′

(
) (
)
xi n + n′hji xj n + n′′ .

(19)n=−L/2

Although the direct calculation of this convolution is computationally complex, equivalent operation is efficiently computed by the
FFT-based overlap-and-add convolution.
Subsequently, for each of h = A, B, we find the interval pairwith higher the correlation between the channels. We select the oneof the index pairs (i, j) = (1, 2) , (2, 1) with the higher coherence|γh (i, j)|2 given by(i∗h , jh∗ ) =|γh (i, j)|2 ,arg max(20)(i,j)=(1,2),(2,1)
2
N
−1
∑

|γh (i, j)|2 =

xi

(

) (
)
n + n′hji xj n + n′′hij

n=− L
2
N
−1
∑

−1
∑
(
)2 N
(
)2
xi n + n′hji
xj n + n′′hij

n=− L
2

n=− L
2

.

Finally, nA1 , nA2 , nB1 and nB2 estimated by the interval pairs withthe higher coherence asnhi∗h = n′hjh∗ i∗h ,(22)= n′′hi∗h jh∗ .

(23)

n

x1 (n )

(Detected appropriately
by coherence evaluation
22
γ A (2,1) > γ A (1,2) )

0

n

0

n

x2 (n )

(Rejected by(Impossible
to detectsince n < 0 )nA 2 = nA′ 21nA′′ 21 coherence

evaluation)(Detected appropriately)

Figure 1: Estimation of nA1 and nA2 corresponding to the same
continuous time near the beginning of the asynchronous recording.
The parameters ϵ and D21 are roughly estimated by substituting
these in (15) and (16). Finally, these roughly-estimated parameters
are used in the STFT synchronization discussed in the previous section to obtain.
5. FINE COMPENSATION OF SAMPLING FREQUENCYMISMATCH
Here we discuss the fine compensation of the sampling frequency
mismatch ϵ. Since the estimation of D21 discussed in the previous
section is reliable, we estimate the parameter ϵ′ to modify the estimation of sampling frequency in the following form keeping themodification of the offset as small as possible.
()
n2 = ϕ′21 n1 ; ϵ′(())= (1 + ϵ) 1 + ϵ′ (n1 − M ) + M − D21 ,(24)
where M denotes the central sample of the whole the duration ofx1 (n) the frame analysis is applied.
Since the rough estimation of ϵ is already obtained, themodification of synchronization by ϵ′ tends to be small. Thus
the fine synchronization is conducted as the linear phase filtering of X̂2 (k, ϕ21 (n1 )) in (12) to obtain the modified signal
X̂2′ (k, ϕ′21 (n1 ; ϵ′ , M )) as(())
X̂2′ k, ϕ′21 n1 ; ϵ′)(2πȷkϵ′ (n1 − M ).
(25)= X̂2 (k, ϕ21 (n1 )) exp
L
The parameter ϵ′ can be optimized as a maximum likelihood estimate assuming the two-channel signal given by
(
) [
(
(
))]T
X̂′ k, n1 ; ϵ′ = X1 (k, n1 ) , X̂2′ k, ϕ′21 n1 ; ϵ′(26)
is stationary with the correctly estimated mismatch modification ϵ′[4]. The log likelihood is given by
∑
∑ ′(
( )
)
(
)H
L ϵ′ = −log detX̂ k, n1 ; ϵ′ X̂′ k, n1 ; ϵ′ . (27)
n1

k
′

(21)

∗
hjh

nA1 = nA′′ 12

′ 2
nA1

The parameter ϵ can be optimized by the combination of discretized search and golden section search as in [4].
Note that the optimization of ϵ′ is equivalent to the following
update of ϵ and D21 .
()ϵ ← (1 + ϵ) 1 + ϵ′ − 1,
(28)
D21 ←

ϵ′ M + D21.
1 + ϵ′(29)

2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

October 20-23, 2013, New Paltz, NY

1
-2
10 1

Signal length [s]Reverberation time
Frame length L
Frame shift
Source distances
Source directions
Microphone spacing
N
Candidates of
discretized search [4]3, 5, 10, 20, 30, 60, 120, 300 and 600
T60 of 130 ms
4,096 samples
2,048 samples
1.5 m[-50◦ , 30◦ ], [-60◦ , -10◦ ]2 cm
16,000 samples
10][ samples−4from−2 × 10 , 2 × 10−4

RMSE of

Table 1: Experimental conditions.

10

100

1000

Conventional-4

10

Coarse-610estimate
Proposed

-8

10

-10

10

Signal length [s]Figure 2: Root mean squared errors (RMSEs) of estimations ofsampling frequency mismatches ϵ.
25
No mismatch
20
Unprocessed
SDR [dB]

Although the synchronization of X̂′ (k, n1 ; ϵ′ ) in the STFT domain
with the maximum likelihood estimate ϵ′ is sufficiently accurate as
we show in the experiment, it is also possible to apply other resampling methods to synchronize the channels with the estimatedparameters ϵ and D21 in (28) and (29).

15Upper limit of STFT
10synchronization
Conventional

5
Proposed0

6. EVALUATION
To confirm the effectiveness of the proposed blind synchronization,
we gave artificial sampling frequency mismatch to observation of
two speakers’ speech with two microphones, and evaluated the accuracy of the sampling frequency mismatch compensation and its
contribution to BSS.
The observed signals are made by convolution of measured impulse responses and speech signals, which are generated by concatenation of Japanese word utterances. We evaluated all the 12 combinations of two speakers from two male and two female speakers.
The original sampling frequency of the observation is 16,000 Hz,
and to one channel we gave modifications of sampling frequency of±0.5, ±1, ±1.5 Hz, which are realistic as practical bias of sampling frequencies. To generate the artificial sampling frequency
mismatch, we used resampling with the polyphase filters. We usedauxiliary-function-based independent vector analysis [6] to conduct
BSS. Other conditions are listed in Table 1.
To examine accuracy of the estimation of sampling frequency
mismatch ϵ, we compared the root mean squared errors (RMSEs) of
the conventional method [4], the coarse search discussed in Sect. 4
and the proposed method in Fig. 2. It can be clearly seen that theestimation accuracy improves with the increase of the observation
length. The estimation accuracy is improved from the coarse estimate by the proposed method. The conventional and the proposed
methods performs similarly.
We compared the source separation performance in Fig. 3. The
evaluation criterion is signal-to-distortion ratio (SDR) [7]. The
curve labeled as no mismatch shows the performance of the synchronized recording without the artificial mismatch, and its SDRs
are highest. We can see the very low performance of the unprocessed signal and compensation of synchronization is necessary inthis condition. The performance of the conventional method degrades the performance with the observation longer than 30 s evenwith the high accuracy similar to the proposed method despite theaccurate estimation. Thus the optimization of frame analysis is necessary with such the long data. We evaluated the upper limit of the
STFT synchronization by substituting true values of ϵ and D21 in(12), which shows small degradation of about 1 dB from the no mis-

1

10

100

1000

Signal length [s]Figure 3: Source separation performance with SDR measure.

match. The performance of the proposed method has little degradation from the upper limit. Note that such the small error is kept
with the observation longer than 600 ms. Therefore it is confirmedthat the proposed synchronization successfully recovers the source
separation performance.
7. REFERENCES[1] Z. Liu, “Sound source separation with distributed microphonearrays in the presence of clock synchronization errors,” Proc.
IWAENC, 2008.
[2] R. Lienhart, I. Kozintsev, S. Wehr and M. Yeung, “On the importance of exact synchronization for distributed audio processing,” Proc. ICASSP, pp. 840–843, 2003.
[3] S. Markovich-Golan, S. Gannot and I. Cohen, “Blind sampling
rate offset estimation and compensation in wireless acoustic sensor networks with application to beamforming,” Proc.
IWAENC, 2012.
[4] S. Miyabe, N. Ono and S. Makino, “Blind compensationof inter-channel sampling frequency mismatch with maximum likelihood estimation in STFT domain,” Proc. ICASSP,
pp. 674–678, 2013.
[5] S. Makino, T.-W. Lee and H. Sawada, Eds., Blind Speech Separation, Springer, 2007.
[6] N. Ono, “Stable and fast update rules for independent vector
analysis based on auxiliary function technique,” Proc. WASPAA, pp. 189–192, 2011.
[7] E. Vincent, H. Sawada, P. Bofill, S. Makino and J. Rosca, “First
stereo audio source separation evaluation campaign: data, algorithms and results,” Proc. ICA, pp. 552–559, 2007.