VMInNet: Interpolation of Virtual Microphones in Optimal Latent Space
Explored by Autoencoder
Riki Takahashi, Li Li, Shoji Makino, Takeshi Yamada
University of Tsukuba, Japan
E-mail: {r.takahashi@mmlab.cs, lili@mmlab.cs, maki@tara, takeshi@cs}.tsukuba.ac.jp

1. Introduction
In recent years, with the development of automatic speech
recognition (ASR) and robot hearing, the importance ofspeech enhancement has considerably increased. Owing to
the wide usage of stereo microphone built-in small devices
such as smartphones and voice recorders, speech enhancement that serves dual-channel signals is a particular requisite.
Beamforming and blind source separation (BSS) are the twomain methods to deal with this problem. Beamforming and
BSS methods using spatial filtering [1, 2, 3] are noteworthy
in the low distortion of the enhanced target speech. However, their performance degrades when there are fewer microphones than sources, i.e., underdetermined conditions. To
achieve satisfactory speech enhancement performance withsuch devices having small microphone array, many methods
have been proposed to enhance speech in underdeterminedsituations such as time-frequency masking [4, 5, 6], multichannel Wiener filtering [7, 8], and nonnegative matrix factorization (NMF) [9]. Although these methods are noteworthy that they can significantly improve speech intelligibility
in underdetermined situations, they face a tradeoff between
low signal distortion and high noise reduction performance.

virtual mic

virtualsignalreal
signalSignalprocessing

realsignalreal mic 2

interpolation

Abstract
In this paper, we propose a new method for interpolationof virtual signals between two real microphones to improve
speech enhancement performances in underdetermined situations. The virtual microphone technique is a recently proposed technique that can virtually increase the channel of observed signals by linearly interpolating the phase and nonlinearly interpolating the amplitude based on β-divergencein the short-time Fourier transform (STFT) domain. This
technique has been shown to be effective in improving the
speech enhancement performance of beamforming in underdetermined situations. It is reasonable to linearly interpolate
the phase based on the sound propagation model and nonlinearly interpolate the amplitude to increase the information
content of the observed signals. However, there is no theoretical proof that β-divergence is the optimal criterion for
amplitude interpolation due to the complexity of the physicalmodel of amplitude. In this paper, we propose using an autoencoder to search for the optimal interpolation domain in adata-driven manner. We perform amplitude interpolation in
the latent space, a low dimensional representation space ofobserved mixture signals that is trained so that the interpolated virtual signals are optimal for conducting beamformingwith high performance. The experimental results revealed
that the proposed method achieved higher speech enhancement performance than the conventional methods.

real mic 1

Figure 1: Microphone array signal processing with virtual
microphone technique.
On the other hand, the virtual microphone technique [10]allows the well-studied determined methods to be applied tothe signals recorded in underdetermined situations by virtually increase the number of channels. With two real microphone signals, the virtual microphone technique estimates theobserved signal at a position where is no real microphone
placed by interpolating phase and amplitude independently.
Based on the W-disjoint orthogonality (W-DO) [11, 4] assumption, phases of virtual signals can be obtained using
linear interpolation by approximately modeling propagatingwaves as plane waves. For amplitude estimation, since modeling the amplitudes of propagating waves is difficult due tothe complicated acoustic environments, complex logarithmic
interpolation and a generalized version, where the interpolation rules are derived as closed-form solutions of an optimization problem formulated using β-divergence, have been proposed and experimentally shown to be effective in improving
speech enhancement performances using a maximum signalto-noise (maxSNR) beamformer. Furthermore, the results inpaper [10] indicated that speech enhancement performances
tend to increase when the improved nonlinearity is applied toamplitude interpolation.
However, there is no theoretical proof that β-divergenceis the optimal criterion for formulating the amplitude interpolation problem. Because the physical model of amplitude
is complicated, it is challenging to design appropriate rules
manually. To overcome this problem, this paper proposes
using an autoencoder to search for the optimal interpolation93

domain in a data-driven manner. Specifically, we perform
amplitude interpolation in the latent space and train the autoencoder so that the interpolated virtual signals become optimal for conducting beamforming. We use 2-dimensional
fully convolutional neural networks (CNNs) to design the autoencoder, which allows the network to handle inputs with arbitrary length and perform interpolation by taking the whole
spectrogram into account. This is different from the conventional method using β-divergence, where the interpolation is
performed in a time-frequency bin-wise manner.
Related work: An attempt has already been made to employ a CNN to estimate the amplitude of virtual signals [12],where a CNN is trained to output the amplitude directly given
the amplitudes of the observed signals and the interpolation
position. This paper differs from this method by training an
autoencoder to constrain the estimated amplitude to be a mixture signal by explicitly performing interpolation in the latent space and then passing it through the decoder, which istrained as a reconstructor of the mixture signals. This design makes the method more intuitive and the network moreinterpretable. To overcome the strong assumption made forphase and amplitude interpolation, a method that trains atime-domain TasNet to estimate virtual signals in the time
domain has been recently proposed [13]. Although this supervised method has shown high performance in improving
speech enhancement performance, training networks using
real microphone signals limits the flexibility of location toplace the virtual microphone. Moreover, collecting training
data in various acoustic conditions are high-cost.
2. Virtual microphone technique based on β-divergence
We model the microphone signals in the STFT domain.
Here, let xm (ω, t), m = 1, 2 be the mth real microphonesignal at the angular frequency ω in the tth time frame.
The amplitudes of these signals are denoted as Am (ω, t) =|xm (ω, t)| and the phases are denoted as ϕm (ω, t) =∠xm (ω, t). A virtual microphone signal v(ω, t, α, β) is defined as an observation at the point α obtained by internallydividing the line joining two real microphones in the ratio
α : (1 − α) (See Fig 1). β is the hyperparameter of βdivergence, which is used as the metric for amplitude interpolation. Hereafter, we omit ω, t, α, and β for notation simplicity.
We first introduce the interpolation of phase. We assume
W-DO [4, 11] for mixed signals that each time-frequency bin
is dominated by at most one sound source, which means thatthe observed signal in each time-frequency bin can be regarded as a single wave. Based on this assumption, the physical model of propagating waves can then be approximated asthat of a plane wave. The phase ϕv = ∠v of a virtual microphone signal v can then be interpolated linearly on the basisof that model assources and microphones, and direction of arrival (DOA) of
sources, it is difficult to faithfully model the amplitude of a
propagating wave. Therefore, an interpolation rule based on
the β-divergence has been used instead of considering physical models [10]. The β-divergence between the amplitude ofa virtual microphone Av = |v| and that of the ith real microphone Ai is defined as
Dβ (Av , Ai ) =
Av (log Av − log Ai ) + (Ai − Av )
 Av − log Av − 1
Ai
Ai

β

A
Aβ Av Aβ−1

v
i

+ i −

β(β − 1)ββ−1(β = 1),(β = 0),(otherwise),(3)

where Dβ (Av , Ai ) is continuous at β = 0 and β = 1. Theinterpolation rule of the amplitude Av is then given as theclosed-form solution of the optimization problem that minimizes σDβ , the sum of Dβ (Av , Ai ) weighted by the hyperparameter of the virtual microphone interpolation position α:σDβ = (1 − α)Dβ (Av , A1 ) + αDβ (Av , A2 ),Av = argminAv σDβ .

(4)
(5)

By differentiating σDβ with respect to Av and setting it to 0,
the interpolated amplitude is obtained as
Av =exp ((1 − α) log A1 + α log A2 ) (β = 1),() 1β−1 β−1 (1 − α) Aβ−1
(otherwise).
+
αA
1
2

(6)

Note that the phase can be interpolated with arbitrary real
number α, whereas the amplitude interpolation is defined
only in the domain of 0 ≤ α ≤ 1 when β ̸= 1. The extrapolation of a virtual microphone in the domain α < 0 and
α > 1 was considered in [14].
It has shown that the virtual microphone technique basedon the β-divergence is effective in improving speech enhancement performance in underdetermined situations [10]. However, there is no proof that β-divergence is the optimal criterion for amplitude interpolation since we cannot analyze thephysical model of amplitude.

3. Proposed method: VMInNet
In this paper, we propose using an autoencoder to search
for the optimal interpolation domain in a data-driven manner
to overcome the above-mentioned problem. The flowchart ofthe proposed method is shown in the Fig. 2. In the proposed
method, the two-channel observed signals are separated into
ϕv = (1 − α) ϕ1 + αϕ2 .
(1) the amplitude spectrograms Am and phase spectrograms ϕmas the conventional method does. The amplitude spectrogram
Since the observed phase has an aliasing ambiguity given by Am is then embedded into latent variables zm = Eθ (Am ) by
ϕi ± 2ni π with integer ni , we need to make a constraint
an encoder network Eθ (·). We interpolate the amplitude ofvirtual microphone signal at the latent space as|ϕ1 − ϕ2 | ≤ π(2)zv = (1 − α)z1 + αz2 .
(7)
to eliminate the ambiguity of the interpolated phase.
For the interpolation of amplitude, since there are many The latent variables z1 , z2 , zv are then converted to the amacoustic conditions such as the distance between the sound plitudes Am = Dψ (zm ) and Av = Dψ (zv ) by a decoder

94

ଶ

௩

interpolation

𝑤 513 ∗ 128 257 ∗ 64 129 ∗ 32 65 ∗ 16 33 ∗ 8 17 ∗ 4

Figure 2: Flowchart of proposed method.

output (𝐴 , 𝐴 , 𝐴 )input (𝐴 , 𝐴 )ଶenhanced signal

௩

concatenate௩

decoderinterpolation by
VMInNetଵ

speech enhancement with MPDR

ଵlinearinterpolation௠

encoderଶ

angle

observed signal

௠

abs

ଵ

decoder

encoder

𝑐

1

128

256

512

1024

𝑘

5∗5

5∗5

5∗5

5∗5

5∗5

2048

𝑤 17 ∗ 4 33 ∗ 8 65 ∗ 16 129 ∗ 32 257 ∗ 64 513 ∗ 128
𝑐 2048 1024

512

256

128

𝑘 5∗4 5∗4

5∗4

5∗4

5∗4

1

Figure 3: Network architecture of VMInNet. “w”, “c”, and“k” denote feature size, channel number, and kernel size, respectively.

network Dψ (·). Here, θ and ψ denote trainable parametersof the encoder and decoder networks. The phase of virtualsignal ϕv is linearly interpolated using (1). The virtual signal acoustic transfer functions h(ω) = [h1 (ω) · · · hM (ω)]T fromv is obtained by concatenating the estimated amplitude and the target source to the microphone array, i.e., a(ω) =[]Tphase spectrogramhM (ω)
1 hh21 (ω).
(ω) · · · h1 (ω)
v = Av exp{jϕv }.
(8)
The encoder and decoder parameters θ and ψ are trained
We call the network “VMInNet”, an abbreviation of “interpo- by minimizing the following loss function:
lation network for virtual microphone”.
∑
To maximize the potential of the data-driven method,L(θ, ψ) =|Dψ (Eθ (Am (ω, t))) − Am (ω, t))|2
where task-dependent loss function is allowed, we trainm,ω,t
the network using minimum power distortionless response∑(MPDR) beamformer-based loss function. Specifically, the|wH (ω)x(ω, t) − s(ω, t)|2 . (13)+loss function is designed to minimize the mean squaredω,terror between the target signal s(ω, t) and the output of
the MPDR beamformer to enforce the generated ampli- Here, the first term of (13) is the restoration error of the autude to be optimal for constructing the MPDR beam- toencoder to ensure that the virtual amplitude latent variables
former. For the observed signal consisting of two real generated in the latent space are restored to the amplitude
and I virtual microphone signals, we use x(ω, t) = space. The second term is the MPDR beamformer loss. ThisT
[x1 (ω, t), v1 (ω, t), · · · , vI (ω, t), x2 (ω, t)] to denote the training criterion allows the network to search for the optimixture signal vector. A beamformer that enhances the mal interpolation space where the interpolated virtual signal
source of interest is given by
is optimal for constructing an MPDR beamformer. Note that
other task-dependent loss function can also be considered asHy(ω, t) = w (ω)x(ω, t),(9) long as the loss function is differentiable.
T
The network architecture is shown in Fig. 3. We use 2w(ω) = [w1 (ω) · · · wM (ω)] ,(10)dimensional CNNs to carefully design the network architecwhere y(ω, t) is the output signal of the beamformer, w(ω) ture, which allows the network to handle inputs with arbitrary
length and perform interpolation by taking the whole spectrodenotes the spatial filter vector, (·)T denotes the transpose, gram into account. After each CNN layer, we employ batchH(·) denotes the Hermitian transpose, and M = 2 + I de- normalization to stable the training process.
notes the number of channels of the observed signals. The
spatial filter w(ω) derived based on the MPDR beamformer
4. Experiments
is expressed asw(ω) =

Φ(ω)−1 a(ω)
aH (ω)Φ(ω)−1 a(ω)

,

(11)

4.1 Experimental conditions

To evaluate the effectiveness of the proposed method, we(12) conducted speech enhancement experiments in underdetermined situations. The training dataset consisted of 3 speakers
Here, Φ(ω) is a covariance matrix of the observed sig- excerpted from the Wall Street Journal (WSJ0) corpus [15]nals at frequency ω, and a(ω) is the relative transfer func- and the audio files for each speaker were 1 minute. The testtion (RTF) of target, which is defined as the ratio of the dataset was comprised of 10 speakers, where the audio files
Φ(ω) = E[x(ω, t)x(ω, t)H ].

95

Table 3: SDR, SIR, SAR [dB] achieved with each method.
Conditions
SDR
SIR SARamp (prop), phase (linear), tf (beta) 7.04 13.32 8.36
7.38 13.98 8.75
amp (prop), phase (real), tf (beta)
amp (prop), phase (linear), tf(real)
7.37 13.94 8.74
7.41 14.02 8.77
amp (prop), phase (real), tf (real)

Table 1: Experimental conditions
Number of real microphones M
2 or 3
Number of sound sources N
3
4 cm
Distance between microphones
Reverberation time
120 msSampling rate
8 kHz
Input SNR
0 dB
Window length / shift
1024 / 512 samples
Table 2: SDR, SIR, SAR [dB] achieved with each method.
Conditions
SDR
SIR
SAR
2 real mic + 1 vir mic (beta)
5.97
8.58 10.06
2 real mic + 1 vir mic (cnn)
5.34 15.54
5.92
2 real mic + 1 vir mic (prop.)
7.04 13.32
8.36
2 real mic
2.18
2.86 12.56
3 real mic
16.07 19.49 18.85

signals, which aims to improve speech enhancement performances in underdetermined situations. We trained the network to search for the interpolation domain, which is optimal for conducting an MPDR beamformer. The experimental
results revealed that the proposed method outperformed theconventional virtual microphone techniques by achieving a
SDR improvement of about 1.1 dB.

Acknowledgments
This work was partly supported by JSPS KAKENHI Grant
Numbers JPH04131 and JP19J20420.

for each speaker were about 9 minutes. The observed mixture signals were generated by adding reverberant signals of
3 speakers together, where the reverberant signals were generated by convolving the impulse responses simulated by aroom impulse response generator [16] with clean speech signals. The DOA of the target was set to 90◦ , and those of interferers were set to 50◦ and 150◦ with reverberation time of 120
ms. The experimental conditions are listed in Table 1. We
used signal-to-distortion ratio (SDR), signal-to-interference
ratio (SIR) and signal-to-artifacts ratio (SAR) [17] as evaluation criteria to quantify the speech enhancement performance.
Here, we used source image of signals as reference signals.
We chose the method interpolating amplitude based on βdivergence (beta) [10], and that uses a CNN to directly estimate the amplitude of virtual signals (cnn) [12] as baseline
methods. We assumed that RTF was known for all the methods. The RTF at the position of virtual microphone is estimated using the conventional β-divergence-based method.
The RTF can also be estimated using a neural network, which
is one direction of our future work since we would like tofocus on amplitude interpolation in this work.

References[1] P. Smaragdis, “Blind separation of convolved mixtures in the frequency domain,”
Neurocomputing, vol. 22, no. 1-3, pp. 21–34, 1998.
[2] D. Kitamura et al., “Determined blind source separation unifying independentvector analysis and nonnegative matrix factorization,” IEEE/ACM TASLP, vol.
24, no. 9, pp. 1622–1637, 2016.
[3] T. Higuchi et al., “Online MVDR beamformer based on complex Gaussian mixture model with spatial prior for noise robust ASR,” IEEE/ACM TASLP., vol. 25,
no. 4, pp. 780–793, 2017.
[4] O. Yılmaz et al., “Blind separation of speech mixtures via time-frequency masking,” IEEE TSP, vol. 52, no. 7, pp. 1830–1847, 2004.
[5] S. Rickard, “The DUET blind source separation algorithm,” in Blind Speech
Separation, S. Makino et al., Eds., pp. 217–241, Springer, 2007.
[6] H. Sawada et al., “Underdetermined convolutive blind source separation via frequency bin-wise clustering and permutation alignment,” IEEE TASLP, vol. 19,
no. 3, pp. 516–527, 2010.
[7] N. Q. K. Duong et al., “Under-determined reverberant audio source separation
using a full-rank spatial covariance model,” IEEE TASLP, vol. 18, no. 7, pp.
1830–1840, 2010.
[8] R. V. Rompaey et al., “GEVD based speech and noise correlation matrix estimation for multichannel wiener filter based noise reduction,” in Proc. EUSIPCO, pp.
2562–2566, 2018.

4.2 Results and discussion
Table 2 shows the results obtained by each method. The
proposed method achieved the best score in terms of SDR,
which confirmed the effectiveness of the proposed method in
improving speech enhancement performance. This result indicates that interpolation in latent space is more effective thanusing β-divergence and directly estimating interpolated amplitudes with a CNN. We also compared the results of using
phase or RTF of real microphones and VMInNet for amplitude interpolation to investigate the importance of accuratelyinterpolating amplitude. From these results, we found that
although accurate phase and RTF could improve the performance, the improvement was slight, which indicates that theaccuracy of amplitude highly influences the speech enhancement performance.
5. Conclusion
In this paper, we proposed VMInNet, an autoencoder network for interpolating the amplitude of virtual microphone[9] A. Ozerov et al., “Multichannel nonnegative matrix factorization in convolutive
mixtures for sudio source separation,” IEEE TASLP, vol. 18, no. 3, pp. 550–563,
2010.
[10] H. Katahira et al., “Nonlinear speech enhancement by virtual increase of channels
and maximum SNR beamformer,” EURASIP JASP, vol. 2016, no. 1, pp. 1–8, Jan.
2016.
[11] A. Jourjine et al., “Blind separation of disjoint orthogonal signals: demixing nsources from 2 mixtures,” in Proc. ICASSP, pp. 2985–2988, 2000.
[12] K. Yamaoka et al., “Cnn-based virtual microphone signal estimation for mpdr
beamforming in underdetermined situations,” Proc. EUSIPCO, pp. 1–5, 2019.
[13] T. Ochiai et al., “Neural network-based virtual microphone estimator,” arXiv
preprint arXiv:2101.04315, 2021.
[14] R. jinzai et al., “Microphone position realignment by extrapolation of virtual
microphone,” in Proc. APSIPA, pp. 367–372, 2018.
[15] Garofolo et al., “Csr-i (wsj0) complete ldc93s6a,” Web Download. Philadelphia:
Linguistic Data Consortium., 1993.
[16] E. A. P. Habets, “Room impulse response (RIR) generator,” Available at:https://www.audiolabs-erlangen.de/fau/professor/habets/software/rir-generator,
2008.
[17] E. Vincent et al., “Performance measurement in blind audio source separation,”
IEEE TASLP, vol. 14, no. 4, pp. 1462–1469, 2006.

96