Proceedings of 2022 APSIPA Annual Summit and Conference

7-10 November 2022, Chiang Mai, Thailand

Encoder Re-training with Mixture Signals on
FastMVAE Method
Shuhei Yamaji∗ , Taishi Nakashima∗ , Nobutaka Ono∗ , Li Li† and Hirokazu Kameoka†∗ Tokyo Metropolitan University, Tokyo, Japan† NTT Communication Science Laboratories, Kanagawa, Japan.

and convergence of the likelihood function in the update are
guaranteed.
However, each iterative update requires a long processing
time for backpropagation. This makes MVAE unsuitable forreal-time processing. To address this, a fast algorithm called
the FastMVAE [11] has subsequently been proposed. The
idea is to replace the process of updating the latent variables
with the forward propagation of the trained encoder. Thisreplacement eliminates the time-consuming backpropagation
steps and significantly reduces processing time. However,
the encoder output is only an approximation of the update
destination that decreases the objective function. Therefore, theobjective function is not guaranteed to monotonically decrease
and converge. For this reasons, the literature [11] reports thatwhile FastMVAE is indeed faster than MVAE, it falls short of
MVAE in separation performance.
To solve this problem, we propose a method to train theencoder to find latent variables that decrease the objectivefunction during the source separation algorithm, while following the idea of updating latent variables using only theforward propagation of the encoder. Specifically, after training
the network (encoder and decoder) in conventional FastMVAE,
only the encoder is re-trained using the likelihood function as
the loss function. Experimental results show that the network
trained by the proposed method reduces the decrease of thelikelihood function and achieves higher separation performance than the conventional method.

Abstract—In this paper, we propose a new network training to
improve the source separation performance of the fast multichannel variational autoencoder (FastMVAE) method. The FastMVAE
method is very effective for supervised source separation. It
also significantly reduces the processing time by replacing the
backpropagation steps in the MVAE method with a single forward
propagation of the encoder for estimating latent variables. In
previous studies, the encoder is trained together with the decoder
using clean speech. In contrast, in this study, we re-train only
the encoder by using the mixed signals with the decoder fixed.
More specifically, using the imperfectly separated signals obtainedin the process of the source separation algorithm, we train theencoder to find the optimal latent variables that minimize the objective function for source separation. Experimental results showthat the proposed method reduces the objective function at almost
every iteration and achieves higher separation performance thanthe conventional method.

I. I NTRODUCTION
Multichannel blind source separation is a promising technique to extract each source signal from a mixture. It has
been remarkably developed in the last two decades [1], andfrequency-domain independent component analysis (ICA) [2],independent vecotr analysis (IVA) [3], [4], auxiliary function
based IVA (AuxIVA) [5], and independent low-rank matrix
analysis (ILRMA) [6] have been proposed. Some of thesemethods have a source model representing a source signal, andthe parameters of the source model and the demixing matrix
are alternately updated to reduce the same objective function.
Recently, multichannel BSS methods using deep neuralnetworks (DNNs) has been actively investigated [7]–[9]. The
high expressive power of DNNs suggests that they may beeffective in separating speech signals. In particular, the multichannel variational autoencoder (MVAE) [10], which introduces a variational autoencoder (VAE) as a source model, iseffective in supervised source separation tasks. MVAE is amethod that uses the decoder distribution of Conditional VAE(CVAE) as a source model for each sound source. CVAE istrained with the spectrogram of a single speaker signal andthe corresponding speaker ID as condition class variables. This
training allows CVAE to be used as a speech source model forthe speakers in the training data. During source separation,
iterative projection (IP) updates the demixing matrix. And,
backpropagation of the CVAE decoder is used to update thesource model, searching for latent variables that maximize theindependence of the separated signals. In this time, if the step
size can be adjusted appropriately, monotonic non-decreasing

II. B LIND S OURCE S EPARATION
A. Problem formulation
Let N be the number of sound sources and microphones.
The speech signal, the mixed signal, and the separated signal
at each time-frequency are denoted as follows
T∈ CN ,

(1)

T

N

∈C ,

(2)

T

N

(3)

sf,t = (sf,t,1 , . . . , sf,t,n , . . . , sf,t,N )xf,t = (xf,t,1 , . . . , xf,t,n , . . . , xf,t,N )yf,t = (yf,t,1 , . . . , yf,t,n , . . . , yf,t,N )∈C ,where f = 1, . . . , F , t = 1, . . . , T and n = 1, . . . , N denotethe frequency bins, time frames, and sources or channel indices, respectively. T indicates transposition. Let Sn ∈ CF ×T
be the complex spectrogram matrix of the nth speech signal,
sf,t,n over all time frequencies, and let Xn ∈ CF ×T be
the complex spectrogram matrix of the mixed signal as well.
Assuming that the mixing system is linear time-invariant and is978-616-590-477-3 ©2022 APSIPAAPSIPA ASC 2022

704

Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:36:35 UTC from IEEE Xplore. Restrictions apply.

Proceedings of 2022 APSIPA Annual Summit and Conference

MVAE

representing the category class of the sound source and zn is alatent variable representing the time variation of the spectrum.
Now, for simplicity, let these latent variables be represented
collectively as ξn = {zn , cn }, and the decoder output of CVAE
represents the following source model

Source model

Freq.

Clean
spectrogram

Time
Class label

EncoderClass label

DecoderRn (ξn ) = gn · Dec(ξn , Φ),

FastMVAE

Source modelTime

EncoderClass label

where Dec(ξn , Φ) ∈ R
is the decoder output, and its f, t
2component is σΦ,f,t,n(ξn ). gn is a variable representing thescale. This source model is the same as in (4), representing
the local Gaussian source model as follows
YpΦ (Sn |zn , cn , gn ) =NC (sf,t,n |0, rf,t,n (ξn )), (7)DecoderFig. 1. Overview of MVAE method (upper) and FastMVAE method (lower).

f,t2
rf,t,n (ξn ) = gn · σΦ,f,t,n(ξn ).

(8)

In the following, Rn (ξn ) is called the CVAE source model.
The CVAE source model Rn (ξn ) represents various single
sources in the training dataset by the latent variable ξn .
IP [5] can be used to minimize (5) with respect to Wf as in
ILRMA. Since (5) is differentiable with respect to ξn , ξn canbe updated to decrease (5) using backpropagation. The scaleparameter gn can be updated to minimize (5) under fixed ξnand Wf as follows
H
xf,t |2
1 X |wf,n
.
(9)
gn ←
2
FT
σΦ,f,t,n(ξn )a complex instantaneous mixing in the time-frequency domain,
the separated signal can be estimated by yf,t = Wf xf,t . whereWf = (wf,1 , · · · , wf,N )H ∈ CN ×N is the demixing matrixand H is the Hermitian transpose.
Under the above assumptions, the speech signal sf,t,n isdefined as a random variable following a complex normal
distribution with mean 0 and variance rf,t,n = E[|sf,t,n |2 ]as followsp(sf,t,n |rf,t,n ) = NC (sf,t,n |0, rf,t,n ).

(6)

F ×T

Freq.

Clean
spectrogram

7-10 November 2022, Chiang Mai, Thailand(4)

Assuming that each speech signal (sf,t,n and sf,t,n′ , n ̸= n′ ) is
statistically independent, the negative log-likelihood function
J for the mixed signal can be expressed asX
J (W, V) = − 2T
log | det Wf |f,tB. MVAE and FastMVAE

2) FastMVAE: MVAE updates the parameters Wf , ξn andgn so that the objective function (5) becomes smaller at each
iteration, which has the advantage of guaranteeing convergenceof the objective function to the stopping point. On the other
hand, the huge computational costs when determining where
to update the CVAE source model parameters ξn is an issue.
FastMVAE was proposed to avoid this problem. Note that
we use FastMVAE2 [11], the latest version of the FastMVAE
method, and all descriptions below are based on this version.
FastMVAE decomposes the posterior distributionpθ (zn , cn |Sn ) into a product of two conditional distributions,such as pθ (zn |Sn )pθ (cn |Sn ), and learns qΦ (zn |Sn )and γΦ (cn |Sn ) to approximate each distribution. As aresult, the search for ξn , which is used to apply the
backpropagation method to (5) , can be substituted by
the forward propagation of qθ1 (zn |Sn ) and γθ2 (cn |Sn ).
Therefore, the backpropagation method is not required, andfast source model updating is possible. Let ξˆn denote theoutputs zn (Sn , θ1 ), cn (Sn , θ2 ) of the encoder with weightparameter Θ = {θ1 , θ2 }, and denoteξˆn = Enc(Sn , Θ).
(10)1) MVAE: The MVAE method uses CVAE to represent thesource model in (5). CVAE is trained to match the posterior
distribution pΦ (zn , |Sn , cn ) ∝ pΦ (Sn |zn , cn )p(zn ) derived
from pΦ (Sn |zn , cn ) and qθ (Sn |zn , cn ) as closely as possible
and has the structure shown in the upper part of Fig.1.
θ and Φ denote the encoder and decoder weight parameters,
respectively. Where cn is a time-invariant latent variable vector

The encoder output ξˆn in the FastMVAE method is notdirectly calculated to reduce the objective function of source
separation (5). Also, the input signal to the encoder during
source separation is a mixed signal Xn , or an imperfectly
separated signal Yn during iterative estimation. Thus, one ofthe problems is that these signals are very different from theclean single speech signals in training phase. For the above

f

+

X
f,t,n

H
|wf,n
xf,t |2
log vf,t,n +
vf,t,n(5)!
,where W = {Wf }f is a set of all the separation matrices,V = {vf,t,n }f,t,n is a set of vf,t,n for all sources and timefrequencies. Here, when the variance rf,t,n is unconstrained,
the (5) is computed independently for each frequency f , thus
occurring arbitrariness in the order of the separated signals.
This problem is called the permutation problem and requires
permutation resolution as a post-processing step to obtainthe correct separation signal. Recently, a method has been
proposed to achieve source separation while avoiding the
permutation problem by introducing constraints on the source
model rf,t,n . ILRMA and MVAE methods are examples ofsuch techniques.

705

Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:36:35 UTC from IEEE Xplore. Restrictions apply.

Proceedings of 2022 APSIPA Annual Summit and Conference

FastMVAE

signal |Yn |.2 estimated in the previous step and the source
model Rn (ξn ) of the decoder output. Therefore, the objective
here is to estimate the latent variable ξˆn as follows
ξˆn = argmin DIS (|Yn |.2 | Rn (ξn ))(11)

Source model

Freq.

Clean
spectrogram

Time

Encoder

Transferand fix

Class label

Proposedξn= argmin DIS (|Yn |.2 | gn · Dec(ξn , Φ))ξnX
|yf,t,n |2= argmin2
gn · σΦ,f,t,n
(ξn )
ξn

Source modelFreq.

separated
spectrogram

7-10 November 2022, Chiang Mai, Thailand

f,t

Encodernck tioBa agaop
pr

TimeClass label

Decoder− log

|yf,t,n |2−
1,
2
gn · σΦ,f,t,n
(ξn )

(12)

(13)

where | · |.2 denotes the element-wise square of a matrix, andDIS (·|·) is the Itakura-Saito divergence between the matrix
elements.
Now, we want the encoder to find the best latent variableξˆn from Yn . Note that Yn is an imperfectly separated signal
since it is obtained during the iterative estimation process. By
substituting the expression ξn = Enc(Yn , Θ) into (12), thefunction to be minimized can be written as DIS (|Yn |.2 | gn ·Dec(Enc(Yn , Θ), Φ). Taking the expectation with respect to
Yn , we obtain the following objective function
L = E(Yn ) DIS (|Yn |.2 | gn · Dec(Enc(Yn , Θ), Φ) . (14)Fig. 2. Overview of the proposed method. Note that during re-training in the
proposed method, only the encoder is re-trained through back-propagation. The
decoder is fixed with the parameters trained in the previous stage (training as
FastMVAE method).

reasons, there is no guarantee that the objective function (5)will decrease in the source model update in the FastMVAE
method. It may increase.
III. P ROPOSED METHOD
A. Motivation

Thus, to obtain an encoder that outputs the best latent variableξn for a given Yn , the encoder parameter Θ should bedetermined to minimize the expression (14). We can see that it
is a form of the classical autoencoder (non-variational) training using imperfectly separated signals Yn as training data.
Namely, the difference from the FastMVAE method is that itlearns as AE instead of VAE. In addition, the training data is
not a clean single speech signal but an imperfectly separated
signal obtained during the iterative estimation process.
Although (14) can be seen as AE training using imperfectly
separated signals. our goal, however, is not to have the encoderand decoder reconstruct Yn as it is. Rather, our goal is to let
the encoder and decoder find the closest match to Yn among
the possible spectrograms of a single source. This is similar
to ILRMA, which updates the source model parameters as
to approximate the spectrograms of the imperfectly separated
signals obtained during the iterative process using low-rankmatrices. In ILRMA, if the number of basis spectra becomes
too large, the source model becomes so flexible that it canrepresent any spectrogram, resulting in poor separation performance. By analogy, we consider it desirable to fix the decoder
to the one trained by the FastMVAE method.
Therefore, we propose to train the encoder and decoder firstby the FastMVAE method. Then, while keeping the decoder
parameter Φ fixed, only the encoder parameter Θ is updated
through the backpropagation of the (14). An overview diagramof network training is shown in Fig. 2.

The previous work [11] has reported that the FastMVAE
method runs as fast as the conventional fast BSS methods but
shows slightly worse separation performance than the MVAE
method. The main difference between the MVAE and lies in
the way the latent variable ξn is updated. As explained in
section II-B, the latent variable is updated through the forward
propagation of the encoder in the FastMVAE method. However, the decrease of the objective function is not guaranteedin this process. Therefore, if the encoder can be trained toconsistently find ξn that decreases (5), we expect to achieveboth the high separation performance of the MVAE method
and the fast processing speed of the FastMVAE method.
To achieve this, we propose to re-train only the encoder
using (5) as the loss function after network (encoder anddecoder) training in the conventional FastMVAE method.This
re-training allows the encoder to find the update destinations sothat the objective function (5) is reduced under the constraintthat the decoder distribution can only represent a single speakersignal. This is very similar to the parameters of the source
model in ILRMA. which minimizes the objective function (5)under the constraint that the source spectrogram is represented
as a low-rank matrix. Namely, the re-trained encoder can beinterpreted as corresponding to the update equation for thesource model parameters in ILRMA.
B. Training of encoder parameter ΘConsider updating ξˆn to decrease the objective function (5)as described above. Minimizing the objective function (5) for
the source model parameters can be expressed as the ItakuraSaito divergence between the power spectrum of the separated

C. Model architecture
In this paper, the network structure used in the first andsecond stages is the same as the FastMVAE method.

706

Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:36:35 UTC from IEEE Xplore. Restrictions apply.

Proceedings of 2022 APSIPA Annual Summit and Conference
8m

2.
5

m

7-10 November 2022, Chiang Mai, Thailand
in a circle with a radius of 2.5 m, centered on the red point.
The reverberation time was about 300 ms.
The sampling frequency, STFT length and shift length were
set to 16 kHz, 128 ms and 64 ms, respectively. The amountof improvement in SI-SDR [15] was used as the evaluationindex.

0.1 m

3m
5m

B. Results
The average SI-SDR improvement for each method is shownin Fig. 4. The proposed method showed high separation
performance for all source numbers. It can also be seenthat FastMVAE and the proposed method do not show asignificant decrease like ILRMA in separation performancewith an increase in the number of sources.
The evolution of the objective functions for each method isshown in Fig. 5. ILRMA updates the source model parameters
so that the objective function is always decreasing. Therefore,
the objective function of ILRMA has not increased in Fig. 5.
On the other hand, FastMVAE often shows an increasing
objective function. Although the proposed method is basedon FastMVAE, the increase in the objective function was
suppressed, and the convergence was similar to ILRMA.

3m: Microphone array: Source

Fig. 3. The room size in the simulation was 8 m×5 m×3 m.The microphonearray was arranged in a circle with a radius of 0.1 m centered on the red point.
The sound sources were randomly arranged in a circle with a radius of 2.5 m
centered on the red point.

The encoder and classifier layers consist of convolutional
layers, LayerNormalization (LN) and Sigmoid Linear Unit
(SiLU). The decoder layer consists of an inverse convolution
layer, LN and SiLU.

V. CONCLUSION
In this paper, we proposed a new encoder training method
on FastMVAE to improve the estimation of the latent variables
at each iteration. After training the encoder and decoder by
the conventional FastMVAE method, only the encoder is retrained using imperfectly separated signals to minimize theobjective function of BSS. Experimental results showed thatthe proposed method suppressed the increase of the objectivefunction at iterative updates and improved the source separation performance.

IV. E XPERIMENTS
A. Conditions
To evaluate the performance of the proposed method, we
conducted source separation experiments using speech signals.
In this experiment, we applied BSS to the mixed signals andcompared the separation performance and how monotonicallydecreasing the objective function is. ILRMA and the FastMVAE method were employed as comparison methods. The
number of bases in ILRMA was set to two.
The WSJ0 [12] dataset was used as training data for the FastMVAE and the proposed methods. For the FastMVAE method,
we used single-speaker signals with simulated reverberation by
Pyroomacoustics [13]. The signal length was set to 10 s. Whilefor the proposed method, we first selected multiple speech
signals from the WSJ0 dataset and simulated the mixing. Then,
we applied the FastMVAE method to the mixtures and stored
the imperfectly separated signals at each step from 0 to 30
iterations. They were used for re-training the encoder by the
proposed method.
The test data used the JVS [14] dataset, and the simulated
speech signal of 10 s was used. The number of sources N was
two and three, and the 50 mixed signals to test were createdfor each source number.
Fig. 3 shows the room configuration, microphone and soundsource layout in the simulation. The room size in the simulation was 5 m×8 m×3 m. The microphone array was arranged
in a circle with a radius of 0.1 m, centered on the red point, asshown in Fig. 3. The sound sources were randomly arranged

ACKNOWLEDGMENTS
This work was supported by JST CREST JPMJCR19A3.
R EFERENCES[1] H. Sawada, N. Ono, H. Kameoka, D. Kitamura, and H. Saruwatari,“A review of blind source separation methods: two converging routes
to ILRMA originating from ICA and NMF,” APSIPA Transactions on
Signal and Information Processing, vol. 8, p. e12, 2019.
[2] P. Smaragdis, “Blind separation of convolved mixtures in the frequency
domain,” Neurocomputing, vol. 22, no. 1, pp. 21–34, 1998.
[3] T. Kim, H. T. Attias, S.-Y. Lee, and T.-W. Lee, “Blind source separation exploiting higher-order frequency dependencies,” IEEE/ACM Trans.
Audio, Speech, Language Process., vol. 15, no. 1, pp. 70–79, 2006.
[4] H. Atsuo, “Solution of permutation problem in frequency domain ICA,
using multivariate probability density functions,” in Independent Component Analysis and Blind Signal Separation. Berlin, Heidelberg: Springer
Berlin Heidelberg, 2006, pp. 601–608.
[5] N. Ono, “Stable and fast update rules for independent vector analysis
based on auxiliary function technique,” in Proc. WASPAA, 2011, pp.
189–192.
[6] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari, “Determined blind source separation unifying independent vector analysis
and nonnegative matrix factorization,” IEEE/ACM Trans. Audio, Speech,
Language Process., vol. 24, pp. 1626–1641, 2016.
[7] N. Makishima, S. Mogami, N. Takamune, D. Kitamura, H. Sumino,
S. Takamichi, H. Saruwatari, and N. Ono, “Independent deeply learned
matrix analysis for determined audio source separation,” IEEE/ACM
Trans. Audio, Speech, Language Process., vol. 27, no. 10, pp. 1601–
1615, 2019.

707

Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:36:35 UTC from IEEE Xplore. Restrictions apply.

Proceedings of 2022 APSIPA Annual Summit and Conference

7-10 November 2022, Chiang Mai, Thailand

Number of sources = 2

12

Number of sources = 3
ILRMA
FastMVAE
Proposed

SI-SDR improvement [dB]

10
8
6
4
2
0
2
4

0

5

10

15
Iteration

20

25

30

0

5

10

15
Iteration

20

25

30

Fig. 4. The SI-SDR improvements with iterations

2.6

Number of sources = 2

1e6

Number of sources = 3
ILRMA
FastMVAE
Proposed

Objective function

2.7
2.8
2.9
3.0
3.1
3.2

0

5

10

15
Iteration

20

25

30

0

5

10

15
Iteration

20

Fig. 5. The objective functions with iterations.

[8] A. A. Nugraha, A. Liutkus, and E. Vincent, “Multichannel audio source
separation with deep neural networks,” IEEE/ACM Trans. Audio, Speech,
Language Process., vol. 24, no. 10, pp. 1652–1664, Jun. 2016.
[9] S. Seki, H. Kameoka, L. Li, T. Toda, and K. Takeda, “Underdeterminedsource separation based on generalized multichannel variational autoencoder,” IEEE Access, vol. 7, pp. 168 104–168 115, 2019.
[10] H. Kameoka, L. Li, S. Inoue, and S. Makino, “Supervised determinedsource separation with multichannel variational autoencoder,” NeuralComputation, vol. 31, no. 9, pp. 1891–1914, Sep. 2019.
[11] L. Li, H. Kameoka, S. Inoue, and S. Makino, “FastMVAE2: On improving and accelerating the fast variational autoencoder-based source separation algorithm for determined mixtures,” ArXiv, vol. abs/2109.13496,v1,
2021.
[12] J. S. Garofolo, G. David, P. Doug, and P. David, “CSR-I (WSJ0)
complete LDC93S6A,” Philadelphia: Linguistic Data Consortium, 1993.
[13] R. Scheibler, E. Bezzam, and I. Dokmanic, “Pyroomacoustics: A python
package for audio room simulation and array processing algorithms,” inProc. ICASSP, Sep. 2018, pp. 351–355.
[14] S. Takamichi, K. Mitsui, Y. Saito, T. Koriyama, N. Tanji, and
H. Saruwatari, “JVS corpus: free japanese multi-speaker voice corpus,”
arXiv preprint, 2019.
[15] J. L. Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr – half-baked
or well done?” in ICASSP 2019 - 2019 IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 626–
630.

708

Authorized licensed use limited to: NTT. Downloaded on November 07,2023 at 09:36:35 UTC from IEEE Xplore. Restrictions apply.

25

30