Variational Component Decoder for Source Extraction from Nonlinear Mixture

1 INTRODUCTION . Signal extraction from nonlinear mixture is a recurring yet hard research problem in signal processing and representation learning . To tackle this problem , conventional methods can be divided into two categories . One is Blind Source Separation ( BSS ) that recovers all sources in the mixture ( Comon & Jutten , 2010 ) , the other is to leverage available prior knowledge to extract only the desired components from the mixture ( Leong et al. , 2008 ) . Following the hardness results of nonlinear BSS based on nonlinear Independent Component Analysis ( ICA ) ( Hyvärinen & Pajunen , 1999 ) , many attempts have been made to realize nonlinear ICA and to further connect it with representation learning ( Hyvärinen & Morioka , 2016 ; 2017 ; Khemakhem et al. , 2020 ) . However , in many practical scenarios , only one or few sources are supposed to be retrieved for the concerned application ; for example , heartbeat monitoring , especially that achieved by contact-free approaches ( Ha et al. , 2020 ) , demands the heartbeat signal ( waveform ) to be extracted out of the nonlinear mixture with other signals such as body movements and respiration . Compared with nonlinear BSS , nonlinear source extraction can be far more efficient and effective under these scenarios : on one hand , extracting only the source of interest could significantly reduce the computation cost for the same category of algorithms ( e.g. , deep-learning based signal extraction would be computationally cheaper than deep-learning based signal separation ) . On the other hand , if prior knowledge of the source is available , the training may substantially leverage such knowledge so as to sufficiently improve the ( later ) inference accuracy . Recently , there is a rising interest in investigating disentangled representation learning exploiting deep learning techniques . Different from conventional source separation methods that demand hardcoded rules , a deep learning model aims to learn the underlying generating factors automatically and adaptively . To this end , the frameworks of variational auto-encoder ( VAE ) ( Kingma & Welling , 2013 ) and generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) are widely used , and variants modifying designs and training techniques to enhance disentanglement have also been proposed , such as β-VAE ( Higgins et al. , 2016 ) . However , as the existing proposals mostly focus on disentangling on the representation in image or video data ( Kingma & Welling , 2013 ; Higgins et al. , 2016 ; Fabius & Van Amersfoort , 2014 ; Yingzhen & Mandt , 2018 ) where signal mixture never take place , we can not directly apply the idea of disentangled representation learning to our problem of source extraction ( a special form of disentanglement ) from nonlinear time-series mixture . Though a few results have been obtained for disentangling the sources in nonlinear mixing using deep learning , they fall into the category of BSS ( Hyvärinen & Morioka , 2016 ; Khemakhem et al. , 2020 ) . In this paper , we propose supervised Variational Component Decoder ( sVCD ) as a robust deeplearning framework for source extraction from nonlinear mixture . As illustrated in Figure 1 , sVCD aims to recover the interested source y from observed signals x formed by source signals mixed in a nonlinear manner . In particular , sVCD involves an encoder to analyze the temporal features of the observed signals and hence to encode them to the hidden state ( posterior distributions in a latent space ) , and it then decodes the hidden state to recover the interested source . A modified variational inference is applied to help approximate the intractable posterior and to achieve a disentangled representation in the latent space dedicated to the interested source , guided by the prior knowledge applied to train sVCD . Essentially , while y is available for supervised training , sVCD takes x as the sole input during the testing phase to obtain ŷ . To summarize , our main contributions include : • We prove that nonlinear source extraction can be formulated as modified variational inference and achieved by maximizing a tighter variational lower bound . • We design a novel sequence-to-sequence ( Seq2Seq ) translation architecture ; it leverages the modified variational inference to extract the desired source from nonlinear mixture . • We provide empirical evidence that sVCD outperforms a state-of-the-art method based on extensive evaluations under different nonlinear mixing scenarios . The rest of the paper is organized as follows . Section 2 presents related works . Section 3 introduces the mathematical framework for source extraction from nonlinear mixture . Section 4 presents the details sVCD model . Section 5 introduces the datasets and reports the evaluation results . Finally , Section 6 concludes this paper and points out potential future applications . 2 RELATED WORKS . Numerous algorithms have been developed to solve the BSS problem ( Comon & Jutten , 2010 ) : they all aim to decompose signal mixtures into individual components without knowing information about the source signals or the mixing process . Among them , Non-negative Matrix Factorization ( NMF ) decomposes the signal by using low-rank approximations under the non-negative constraint ( Wang & Zhang , 2012 ) . Independent Component Analysis ( ICA ) assumes that the underlying signals are statistically independent to separate them ( Hyvärinen & Oja , 2000 ) . BSS achieved by masking and re-weighting the frequency spectrograms have also been implemented by Hidden Markov Models ( HMMs ) ( Roweis , 2000 ) and segmentation methods ( Bach & Jordan , 2005 ) . One common assumption of regular BSS is that the mixing process is linear , under which the statistical independence of the underlying sources is a sufficient condition to constrain the linear unmixing function . However , as we extend BSS to nonlinear mixture , the statistical independence of the sources can no longer be treated as a sufficient constraint for the unmixing ( Hyvärinen & Pajunen , 1999 ) . Harmeling et al . ( 2003 ) and Hyvärinen & Morioka ( 2016 ) have proposed to utilize temporal structure in the mixture to overcome this challenge . Furthermore , several properties on the mixed sources , e.g. , autocorrelation ( Sprekeler et al. , 2014 ) , general non-Gaussian temporal dependency ( Hyvärinen & Morioka , 2017 ) , or non-stationarity ( Hyvärinen & Morioka , 2016 ) , can also provide sufficient constraints for solving the problem . Aother broadly related topic is disentanglement representation learning ( Bengio et al. , 2013 ) . The word “ disentanglement ” means decoupling of generating factors . Unlike BSS performing separation in signal space , disentanglement representation learning uncovers the underlying factors of variation in latent space . With the development of deep learning , disentanglement representation learning based on VAEs has gained momentum ( Kingma & Welling , 2013 ) . By augmenting the lower bound formulation with the coefficient that regulates the independence prior , Higgins et al . ( 2016 ) proposes β-VAE , a framework to discover interpretable latent representations automatically . Subsequent works ( Burgess et al. , 2018 ; Rolinek et al. , 2019 ; Chen et al. , 2018 ) have led to significant accomplishments in understanding the capabilities of nonlinear disentanglement in VAEs . Researchers have also been seeking the formal definition of disentanglement in several works ( Ridgeway , 2016 ; Higgins et al. , 2018 ; Eastwood & Williams , 2018 ) . The aforementioned algorithms assume that the sources are completely unknown , however , in many practical cases , we have some prior knowledge available and do not actually have to perform source separation in a completely blind manner . As such , these algorithms can be extended to incorporate the knowledge from the priors . In Calhoun et al . ( 2005 ) , paradigm information was employed to aid ICA analysis , and the results are shown to be more robust to noise and outperform basic ICA . In another work ( Jeong et al. , 2009 ) , the original NMF algorithm for BSS is significantly improved by enforcing a constraint that requires disjointness under the semi-blind denoising framework . Last but not least , Kameoka et al . ( 2018 ) achieves source separation by leveraging conditional VAE ( Sohn et al. , 2015 ) with source class as labels in priors during training . All the aforementioned approaches do not seem to suit our need for extracting specific source ( s ) . 3 VARIATIONAL INFERENCE FOR SIGNAL EXTRACTION . Unlike VAE ( Kingma & Welling , 2013 ) that aims to recover input by finding an efficient encoding in latent space , our sVCD focuses on extracting a specific component from nonlinear mixture . Therefore , if we directly borrow the idea of variational inference from VAE , the variational lower bound will not be as tight . As such , we propose and prove a new lower bound for our problem setup , so as to endow a new mathematical framework for sVCD . Let x be a nonlinear mixture of several independent sources . In order to find a representation of x , Bayesian modeling ( Lee , 1989 ) can be used to encode the beliefs about the processes that generate x from all components into a model M with latent vectors z and parameters θM . The model is then “ learned ” by inferring z and adjusting the parameters θM . In other words , given the observed mixture x , Bayesian modeling aims to infer the posterior p ( z|x ) . However , since the posterior distribution is intractable , an optimization approach , which is called variational inference ( Blei et al. , 2017 ) , can be leveraged to approach the problem . To be specific , a surrogate distribution q ( z ) is employed to approximate p ( z|x ) by minimizing their KL divergence ( Van Erven & Harremos , 2014 ) . However , the KL divergence itself still involves the intractable posterior p ( z|x ) . To cope with this issue , the KL divergence is decomposed into : KL ( q ( z ) ‖p ( z|x ) ) = log p ( x ) − Eq ( z ) [ log p ( x , z ) q ( z ) ] . ( 1 ) Since the marginal log-likelihood log p ( x ) is independent of the variational distribution q ( z ) , the KL divergence can be minimized by maximizing the variational lower bound Eq ( z ) [ log p ( x , z ) q ( z ) ] . As a particular type of variational inference method , VAE uses deep neural networks to approximate the generative models . It should be noted that in VAE , the distribution q is conditioned on the observation x , and approximated by an encoder network q ( z|x ) . As such , the variational lower bound of VAE can be expressed as : Eq ( z|x ) [ log p ( x , z ) q ( z|x ) ] = Eq ( z|x ) [ log p ( x|z ) ] −KL ( q ( z|x ) ‖p ( z ) ) , ( 2 ) By slight modifications ( Higgins et al. , 2016 ; Kim & Mnih , 2018 ) , the latent representation z of VAE shows disentanglement properties , suggesting that a regular VAE can be used for separating mixed sources . However , unlike the scenario faced by VAE which is unsupervised and source-agnostic , our task is not completely “ blind ” , i.e. , prior knowledge about a specifically interested component y is available . In this case , the variational lower bound should be Eq̃ ( z|y ) [ log p ( y , z ) q̃ ( z|y ) ] , where q̃ ( z|y ) is an approximate posterior distribution in the form q̃ ( z|y ) = ∫ p ( z|x ) q ( x|y ) dx . It can be observed that the posterior q̃ ( z|y ) takes into account the effect of the mixing process q ( x|y ) , thus covering a broader class of distributions and becoming more expressive . We also notice that there is still the desired component y in the denominator . Considering the goal of extraction signal component from a nonlinear mixture , we would like to maximize Eq̃ ( z|y ) [ log p ( y , z ) q ( z|x ) ] instead , in which case x and y should be used as the input and output of the network , respectively . Therefore , we need to study this new lower bound specifically . Lemma 3.1 . We get a tighter lower bound for the source extraction problem : log p ( y ) ≥ Eq̃ ( z|y ) [ log p ( y , z ) q ( z|x ) ] ≥ Eq̃ ( z|y ) [ log p ( y , z ) q̃ ( z|y ) ] . ( 3 ) Proof . For the left inequality , by using Jensen ’ s inequality , we have : Eq̃ ( z|y ) [ log p ( y , z ) q ( z|x ) ] = ∫ q̃ ( z|y ) log p ( y , z ) q ( z|x ) dz = Eq ( x|y ) Eq ( z|x ) [ log p ( y , z ) q ( z|x ) ] ≤ logEq ( x|y ) Eq ( z|x ) [ p ( y , z ) q ( z|x ) ] = log p ( y ) . ( 4 ) For the right inequality , by using Jensen ’ s inequality , we have : Eq̃ ( z|y ) [ log p ( y , z ) q ( z|x ) ] = Eq̃ ( z|y ) [ log p ( y , z ) ] − Eq̃ ( z|y ) [ log q ( z|x ) ] ≥ Eq̃ ( z|y ) [ log p ( y , z ) ] − Eq̃ ( z|y ) [ q̃ ( z|y ) ] = Eq̃ ( z|y ) [ log p ( y , z ) q̃ ( z|y ) ] . ( 5 ) As a result , we can optimize the surrogate lower bound Eq̃ ( z|y ) [ log p ( y , z ) q ( z|x ) ] of sVCD , which is tighter than that of VAE , for the task of source extraction . This variational inference can be readily implemented by the sVCD network in Section 4 .

Summary: Authors propose a supervised variational component decoder (sVCD) framework that estimates a single source signal sequence from non-linear mixtures. Proposed model relies on a sequence-to-sequence translating variational encoder-decoder architecture, where optimization is performed based on a variational lower bound on the source signal’s data likelihood. Experiments are performed on artificially generated nonlinear sequence mixtures, RF sensing data and an EEG-EOG dataset.