EUSIPCO 2013 1569744343

VIRTUALLY INCREASING MICROPHONE ARRAY ELEMENTS BY INTERPOLATION IN
COMPLEX-LOGARITHMIC DOMAIN†

Hiroki Katahira,‡ Nobutaka Ono, † Shigeki Miyabe, † Takeshi Yamada, † Shoji Makino†‡

University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8577 Japan
National Institute of Informatics, 2-1-2, Hitotsubashi, Chiyoda, Tokyo, 101-8430 Japan
katahira@mmlab.cs.tsukuba.ac.jp, onono@nii.ac.jp,{miyabe, maki}@tara.tsukuba.ac.jp, takeshi@cs.tsukuba.ac.jp
ABSTRACTIn this paper, we propose a new array signal processing technique for an underdetermined condition by increasing thenumber of observation channels. We introduce virtual observation as an estimate of the observed signals at positions
where real microphones are not placed. Such signals at virtual
observation channels are generated by the complex logarithmic interpolation of real observed signals. With the increased
number of observation channels, conventional linear arraysignal processing methods can be applied to underdetermined
conditions. As an example of the proposed array signal processing framework, we show experimental results of speechenhancement obtained with maximum SNR beamformers
modified using the virtual observation.
Index Terms— Linear array signal processing, underdetermined, speech enhancement, virtual observation, beamformer
1. INTRODUCTION
Microphone array signal processing is a speech processing
framework based on signals observed with multiple sensors.
Its typical applications are blind source separation (BSS),
source localization and speech enhancement. Among various
classes of array signal processing, the most established andpowerful approach is linear array signal processing, which includes Independent Component Analysis (ICA) [1] for BSS
and MUltiple SIgnal Classification (MUSIC) [2] for directionof arrival (DOA) estimation. However, many of the linear
array processing methods are effective when the number ofmicrophones equals or exceeds the number of sound sources,
and the performance is often greatly degraded with a larger
number of sound sources. The problem with these methods
is that we need a large-scale microphone array to deal with alarge number of sources, or we cannot apply these methods
to signals observed with few sensors.
The most typical way of solving the underdeterminedarray signal processing problem is to cluster the observed
signals assuming sparseness among sources [3, 4, 5]. Whilethese methods are based on an idea that is slightly different from linear array signal processing, the straightforward
way to realize the underdetermined extension of the linear

1

array signal processing is to increase the number of observation channels by using higher dimensional nonlinear
maps. The conventional way of increasing the number ofobservation channels is to use higher dimensional maps toreproduce higher order statistics, such as higher-order cumulants [6, 7, 8] and higher-order moments [9, 10]. Since
higher-order statistics are useful for the classification problem, the higher-dimensional maps are used to improve the
DOA estimation performance by MUSIC [6, 8, 10]. However, these extensions based on higher dimensional maps forthe analysis of higher-order statistics suffer from distortionin their output signals when these extensions are applied
to speech enhancement or BSS because these maps greatlychange the nature of the signals.
In this paper, we propose an extension of array signal processing by introducing virtual observation which causes lessdistortion and residual noise in the output signal of processing. A virtual observation is obtained as an estimation of thesignals at a point where no sensor is placed. We apply thepseudo increase of the observation channels with the virtual
observation to the maximum SNR beamformer [11, 12], andconfirm the effectiveness of the proposed virtual observation.
2. SPEECH ENHANCEMENT WITH LINEAR
ARRAY PROCESSING
2.1. Speech enhancement by linear filter in STFT domain
Throughout this paper, all the processing is conducted in the
STFT-domain. Let si (ω, t) be the i-th source signal at anangular frequency ω in the t-th frame, and let xj (ω, t) be
the observed signal at the j-th microphone. Signals can bemodeled as[]T
x (ω, t) = x1 (ω, t) , · · · , xM (ω, t)≈N∑ai (ω) si (ω, t) ,

(1)

i=1

ai (ω)=[]T
a1,i (ω) , · · · , aM,i (ω) ,(2)where aj,i (ω) is the transfer function from the i-th source tothe j-th microphone and {·}T stands for the transposition of

a matrix. Speech enhancement by beamforming is conductedby constructing a multi-channel filter given by[]Tw (ω) = w1 (ω) , · · · , wM (ω) ,(3)

to reduce the contamination of non-target sources, wherewn∗ (ω) is a filter for the n-th channel and {·}∗ denotes acomplex conjugation. The enhanced signals y (ω, t) are givenasy (ω, t) = wH (ω) x (ω, t) ,(4)
where {·}H stands for the conjugate transposition of a matrix.
2.2. Performance degradation in underdetermined case
The ideal speech enhancement is obtained when w (ω) is orthogonal to all the transfer functions ai (ω) of the non-target
sources as
wH (ω) ai (ω) =y (ω, t) =0, ∀i ̸= iT ,wH (ω)

N∑w (ω) aiT (ω) siT (ω, t) ,Real Microphone

Fig. 1. Arrangement of real and virtual microphones.
3.2. Derivation of appropriate interpolation domain
based on plane-wave model
We derive the function f where the observation is interpolated to generate the virtual observation from the plane wave
model, which is the simplest model of wave propagation.
In this model, the time difference of arrivals is in a linear
relationship with the inter-microphone distance for any direction of the source. Thus in the STFT domain, the phase∠v (ω, t, α) of the virtually-observed signal should satisfy
the following condition.
∠v (ω, t, α) = (1 − α) ∠x1 (ω, t) + α∠x2 (ω, t),(6)where ∠{·} denotes the phase angle. Note that here weassume that the distance between the real microphones issmaller than half of the wavelength, and the following condition is satisfied without spatial aliasing;i=1=

Virtual Microphone(5)ai (ω) si (ω, t)HReal Microphonewhere the iT -th source is the target. However, such a filter w (ω) does not generally exist with linearly independenttransfer function vectors when the dimension M of the observed signal vectors is smaller than the number N of source
signals. Therefore, the performance of the beamformer degrades in an underdetermined condition.

|∠x1 (ω, t) − ∠x2 (ω, t)| ≤ π.

(8)

(9)

The phase of the complex number is expressed as the imaginary part of the complex logarithmic function as follows
Logx = log |x| + j∠x,(10)where Log stands for the complex logarithmic function. Thus
the linear phase interpolation is operated in Eq. (7) using thefollowing logarithmic function;

3. VIRTUAL OBSERVATION DERIVED FROMCOMPLEX LOGARITHMIC INTERPOLATION
3.1. Virtual observation with interpolation in nonlinear
domainIn this section, we formulate our proposed virtual observation approach. We generate the virtual observation as the estimates of signals observed by a virtual microphone placed at
the point without a real microphone is placed. A virtuallyobserved signal v (ω, t, α) is generated as the estimated observation obtained with a virtual microphone placed at theα : (1 − α) internally dividing point of the positions of two
real microphones (Fig. 1) as an interpolation in the domain of
a function f ;v (ω, t, α) =f −1 ((1 − α) f (x1 (ω, t)) + αf (x2 (ω, t))) , (7)where x1 (ω, t) and x2 (ω, t) denote the complex amplitudesof the actually-observed signals of two microphones. The
choice of the nonlinear function f is the most important factor
in the proposed method for generating a virtual observation asa good estimate.

2f (x) =
f −1 (x) =Logx,
exp (x) ,

(11)
(12)

and the virtually-observed signal v(ω, t, α) is estimated asfollowsv(ω, t, α) = exp ((1 − α) Log (x1 (ω, t)) + αLog (x2 (ω, t)))= exp ((1 − α) log |x1 (ω, t)| + αlog |x2 (ω, t)|+ j ((1 − α) ∠x1 (ω, t) + α∠x2 (ω, t))) . (13)Note that the interpolation of the amplitude in the logarithmic domain is valid when a single plane wave arrives because
plane waves do not decay with distance, and |x1 (ω, t)| =|x2 (ω, t)|.
We discuss the propagation of waves other than the planewave. With a single spherical wave, the interpolation depends on the source direction and it is difficult to interpolate the complex amplitude only with information about theactually-observed signals x1 (ω, t) and x2 (ω, t). However,the spherical wave can be approximated as a plane wave whenthe microphones are closely spaced and the inter-microphone

Observed signals

period ΘI :Observed signals with
virtually-increased channels

Interpolation

・・
・

Array
signalprocessing

λ(ω) =

OutputFig. 2. Block diagram of array signal processing with virtual
observation.
distance is much smaller than the distances between thesource and the microphones. When multiple waves are propagating, it is also difficult to estimate the virtual observation
with x1 (ω, t) and x2 (ω, t) because the observation is thesum of multiple wave fronts. However, by assuming that thesource signals are sparse enough and each time-frequency
slot is dominated by a single source, the virtual observation
is effectively estimated by using the proposed interpolation.
We can estimate an arbitrary number of virtual observation channels. Assume we have Nv virtual microphones between two real microphones at regular intervals. We denotethe observation obtained by these Nv virtual microphones as()
nvn (ω, t) = v ω, t,.
(14)
Nv + 1Let ϕ (x) be the observed signal vector composed of actualand virtual observations:ϕ(x (ω, t)) =[x1 (ω, t) , v1 (ω, t) , · · · ,vNv (ω, t) ,x2 (ω, t)

]T

.
(15)

Since the proposed method brings no other modification thanadding the virtual observation in the signal processing as
Fig. 2, it has a potential to be applied to other kinds of array signal processing techniques such as source localization,
blind source separation, etc. We will investigate them infuture work.

b T (ω)w(ω)
b H (ω)Rbw,
Hbb (ω)RI (ω)w(ω)
b
w

(17)

b T (ω) and R
b I (ω) are the covariance matrices of thewhere Rtarget-only period ΘT and the interference-only period ΘI .
The covariance matrices are estimated with both the actuallyand virtually observed signals as,∑b T (ω) = 1
ϕ(xT (ω, t))ϕ(xT (ω, t))H ,
(18)
R
|ΘT |
t∈ΘT
∑
b I (ω) = 1
R
ϕ(xI (ω, t))ϕ(xI (ω, t))H .
(19)
|ΘI |
t∈ΘI

These covariance matrices include the information of higher
order statistics because virtually-observed signals are obtained by non-linear mapping of observed signals. The filterb (ω) used to maximize power ratio λ (ω) is given as anw
eigenvector corresponding to the maximum eigenvalue of thefollowing generalized eigenvalue problem;b T (ω)w(ω)
b I (ω)w(ω).
bbR= λ(ω)R(20)b (ω) has a scaling
Since the maximum SNR beamformer w
ambiguity [11], we revise the beamformer as:bb (ω) ,w(ω)← bk (ω)w(21)

where bk (ω) is the k-th component of b (ω) given byb(ω) =b x (ω) =Rb x (ω)w(ω)
b
R
,
H
b
b (ω)Rx (ω)w(ω)
b
w

(22)

1∑
H
ϕ (x(ω, t)) ϕ (x(ω, t)) .
T t=1

(23)

T

Then enhanced signal yb (ω, t) is obtained asb H (ω) ϕ (x (ω, t)) .
yb (ω, t) = w(24)

4.2. Regularization of covariance matrix
4. APPLICATION OF VIRTUAL OBSERVATIONAPPROACH TO MAXIMUM SNR BEAMFORMER
4.1. Maximum SNR beamformer
In this section, we apply the proposed virtual observation
technique to a maximum SNR beamformer. The modification is realized simply by substituting the actual observation
vector x(ω, t) with the observation vector ϕ(x(ω, t)) of theactual and virtual channels. We construct a multi-channelb (ω) for actually and virtually observed signals given
filter w
by[]T
b1 (ω) , · · · , w
bNv +2 (ω) .
b (ω) = w
(16)w
b (ω) is designed to maximize the power ratio λ (ω)
The filter w
between the target-only period ΘT , and the interference-only

3

The generalized eigenvalue problem of Eq. (20) tends to generate errors caused by the rank deficiency of covariance mab I (ω) when the number Nv of virtual observation chantrix R
nels is increased. Thus we revise the covariance matrix byadding a regularizerb I (ω) ← R
b I (ω) + εE,R(25)

where E is the unit matrix and ε is an imperceptible nonnegative number.
5. EXPERIMENTS
To confirm the effectiveness of the proposed virtual observation, we conducted speech enhancement experiments andevaluated the performance.

䢶䢰䢷

Table 1. Experimental conditions
Number of real microphones
2
Number of sources3
Number Nv of
virtual observation channels
0–9
Source directions−40◦ , 20◦ , 50◦
Real microphone spacing
2.15 cm
Reverberation time
130 msSampling rate
8 kHz
Frame length1024 samples
Frame shift
256 samples
Signal length
20 secTarget only period ΘT length
5 sec
Interference only period ΘI length
5 sec
Coefficient ε of regularizer
10−6 , 10−8 , 10−10 ,10−12 , 10−14 , 0

䢢

䢶
䢵䢰䢷
䣕䣆䣔䢢䣝䣦䣄䣟

䢵ε䢿䢳䢲䢯䢸

䢴䢰䢷ε䢿䢳䢲䢯䢺

䢴ε䢿䢳䢲䢯䢳䢲

䢳䢰䢷ε䢿䢳䢲䢯䢳䢴

䢳ε䢿䢳䢲䢯䢳䢶
ε䢿䢲

䢲䢰䢷
䢲䢢
䢲

䢳

䢴

䢵
䢶
䢷
䢸
䣐䣷䣯䣤䣧䣴䢢䣱䣨䢢䣘䣫䣴䣶䣷䣣䣮䢢䣕䣫䣩䣰䣣䣮䣵

䢹

䢺

䢳䢳

䢻

䢢

䢳䢲

䣕䣋䣔䢢䣝䣦䣄䣟

䢻ε䢿䢳䢲䢯䢸

䢺ε䢿䢳䢲䢯䢺
ε䢿䢳䢲䢯䢳䢲

䢹ε䢿䢳䢲䢯䢳䢴

䢸ε䢿䢳䢲䢯䢳䢶
ε䢿䢲

䢷
䢶䢢
䢲

䢳

䢴

䢵
䢶
䢷
䢸
䣐䣷䣯䣤䣧䣴䢢䣱䣨䢢䣘䣫䣴䣶䣷䣣䣮䢢䣕䣫䣩䣰䣣䣮䣵

䢹

䢺

䢻

Fig. 5. Evaluation values for maximum SNR beamformers
with virtual observation

5.1. Experimental conditions
We used the 20 sec samples of Japanese and English speech
for the sources. Observed signals were formed by the convolution of measured impulse responses and speech signals. We
used three sound sources and two microphones, which was anunderdetermined condition, then it is difficult to achieve goodspeech enhancement with conventional linear microphone array technique.
The other experimental conditions are shown in Table 1.
To evaluate the performance of the beamformers, we used
objective evaluation criteria, namely the Signal-to-Distortion
Ratio (SDR) and Signal-to-Interference Ratio (SIR) [13].
Higher SDR and SIR values mean better speech enhancement
performance.

Ϯ͘ϭϱĐŵ

ZĞĂůDŝĐƌŽƉŚŽŶĞ

Ϯ͘ϭϱĐŵ

EŽDŝĐƌŽƉŚŽŶĞŽƌ ZĞĂůDŝĐƌŽƉŚŽŶĞ
ZĞĂůDŝĐƌŽƉŚŽŶĞŽƌsŝƌƚƵĂůDŝĐƌŽƉŚŽŶĞ

Fig. 6. Microphone layout for comparing with overdetermined case
8

10

7

9
8

6
7

4

4
3

SIR [dB]

Figure 3 shows spectrograms of the source signal and the observed signal, and Fig. 4 shows spectrograms of enhanced
signals. The spectrograms show that the signal is better enhanced with the virtual observation than without it. Figure
5 shows the SDR and SIR when there were of 0 to 9 virtual
observation channels. Note that the zero in the number of virtual signals in the figure indicates that the beamformer wasprocessed with only the actually-observed signals. The SDR
and SIR values are greatly improved when one virtual observation channel is introduced. We consider this to be because
the underdetermined condition is resolved when one virtual
observation channel. Although the SDR value decreases as
the number of virtual observation channels increases when
there are no regularization of covariance matrices, it does notdecrease when there is a regularizer with a larger coefficient.
Thus we can say that the distortion caused by the rank deficiency of non-target covariance matrices has been reduced.
However, we can observe higher SIR values in when ε is set
at a low value. It is considered that regularization caused errors with the beamformers and degraded there ability to reduce interference. We have also compared speech enhance-

SDR [dB]

55.2. Experimental result and consideration

6
5
4
3

2
2
1

1

0

0

Fig. 7. Evaluation values for maximum SNR beamformers
with real and virtual microphone arrays

ment performance of the virtual microphone array (two real
microphones and one virtual microphone) with the real microphone array (three real microphones) in the same layout
shown in Fig. 6. The comparative result in Fig. 7 shows thatthe virtual observation improves the SIR close to that of real
microphone array. Our proposed virtual observation approach
has improved both SDR and SIR, especially the latter. Thus
we have confirmed the effectiveness of our proposed virtual
observation.

Fig. 3. Spectrograms of source signal (left), observed signal (right)

Fig. 4. Spectrograms of enhanced signal: without virtual observation (left), with virtual observation (right)
6. CONCLUSION
In this paper, we proposed a new array signal processing technique for an underdetermined condition. We introduced a virtual observation as an estimate of the observed signals at positions lacking real microphones are not placed. The virtuallyobserved signals were generated by interpolation in the complex logarithmic domain. We applied our proposed virtual
observation approach to a maximum SNR beamformer andthat were not equipped with real microphones. The performance improved in the underdetermined condition, and we
confirmed the effectiveness of our proposed virtual observation technique.

[6] B. Porat, B. Friedlander, “Direction finding algorithms
based on higher order statistics,” IEEE Trans. Signal
Process., vol. 39, no. 9, pp. 2016–2024, 1991.
[7] P. Chevalier et al., “On the virtual array concept forhigher order array processing,” IEEE Trans. Signal Processing, vol. 53, no. 4, pp. 1254–1271, 2005.
[8] P. Chevalier et al., “High-resolution direction finding
from higher order statistics: The 2q-MUSIC algorithm,”
IEEE Trans. on Signal Processing, vol. 53, no. 4, pp.
2986–2997, 2006.

7. REFERENCES[9] S. Miyabe et al., “Kernel-based nonlinear independent
component analysis for underdetermined blind source
separation,” Proc. ICASSP, pp. 1641–1644, 2008.

[1] S. Makino et al., Blind Speech Separation, Springer,
2007.

[10] Y. Sugimoto et al., “Underdetermined DOA estimation by the non-linear music exploiting higher-order moments,” Proc IWAENC2012, 2012.

[2] R. O. Schmidt, “Multiple emitter location and signal parameter estimation,” IEEE Trans., vol. 34, no. 3, 1986.
[3] O. Yilmaz, S. Rickard, “Blind separation of speech mixtures via time-frequency masking,” IEEE Trans on Signal Processing, vol. 52, no. 7, pp. 1830–1847, 2004.
[4] Y. Izumi et al., “Sparseness-based 2ch BSS using the
EM algorithm in reverberant environment,” WASPAA,
pp. 147–150, 2007.
[5] H. Sawada et al., “Underdetermined convolutive blindsource separation via frequency bin-wise clustering andpermutation alignment,” IEEE Trans. on Audio, Speech& Language Processing, vol. 19, no. 3, pp. 516–527,
2011.

5

[11] S. Araki et al., “Blind speech separation in a meeting situation with maximum SNR beamformers,” Proc.
ICASSP, vol. I, pp. 41–45, 2007.
[12] H. L. Van Trees, Optinum Array Processing, John Wiley& Sons, 2002.
[13] E. Vincent et al., “Performance measurement in blindaudio source separation,” IEEE Trans. on Audio, Speech& Language Processing, vol. 14, no. 4, pp. 1462–1469,
2006.