Variational Component Decoder for Source Extraction from Nonlinear Mixture

1 INTRODUCTION . Signal extraction from nonlinear mixture is a recurring yet hard research problem in signal processing and representation learning . To tackle this problem , conventional methods can be divided into two categories . One is Blind Source Separation ( BSS ) that recovers all sources in the mixture ( Comon & Jutten , 2010 ) , the other is to leverage available prior knowledge to extract only the desired components from the mixture ( Leong et al. , 2008 ) . Following the hardness results of nonlinear BSS based on nonlinear Independent Component Analysis ( ICA ) ( Hyvärinen & Pajunen , 1999 ) , many attempts have been made to realize nonlinear ICA and to further connect it with representation learning ( Hyvärinen & Morioka , 2016 ; 2017 ; Khemakhem et al. , 2020 ) . However , in many practical scenarios , only one or few sources are supposed to be retrieved for the concerned application ; for example , heartbeat monitoring , especially that achieved by contact-free approaches ( Ha et al. , 2020 ) , demands the heartbeat signal ( waveform ) to be extracted out of the nonlinear mixture with other signals such as body movements and respiration . Compared with nonlinear BSS , nonlinear source extraction can be far more efficient and effective under these scenarios : on one hand , extracting only the source of interest could significantly reduce the computation cost for the same category of algorithms ( e.g. , deep-learning based signal extraction would be computationally cheaper than deep-learning based signal separation ) . On the other hand , if prior knowledge of the source is available , the training may substantially leverage such knowledge so as to sufficiently improve the ( later ) inference accuracy . Recently , there is a rising interest in investigating disentangled representation learning exploiting deep learning techniques . Different from conventional source separation methods that demand hardcoded rules , a deep learning model aims to learn the underlying generating factors automatically and adaptively . To this end , the frameworks of variational auto-encoder ( VAE ) ( Kingma & Welling , 2013 ) and generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) are widely used , and variants modifying designs and training techniques to enhance disentanglement have also been proposed , such as β-VAE ( Higgins et al. , 2016 ) . However , as the existing proposals mostly focus on disentangling on the representation in image or video data ( Kingma & Welling , 2013 ; Higgins et al. , 2016 ; Fabius & Van Amersfoort , 2014 ; Yingzhen & Mandt , 2018 ) where signal mixture never take place , we can not directly apply the idea of disentangled representation learning to our problem of source extraction ( a special form of disentanglement ) from nonlinear time-series mixture . Though a few results have been obtained for disentangling the sources in nonlinear mixing using deep learning , they fall into the category of BSS ( Hyvärinen & Morioka , 2016 ; Khemakhem et al. , 2020 ) . In this paper , we propose supervised Variational Component Decoder ( sVCD ) as a robust deeplearning framework for source extraction from nonlinear mixture . As illustrated in Figure 1 , sVCD aims to recover the interested source y from observed signals x formed by source signals mixed in a nonlinear manner . In particular , sVCD involves an encoder to analyze the temporal features of the observed signals and hence to encode them to the hidden state ( posterior distributions in a latent space ) , and it then decodes the hidden state to recover the interested source . A modified variational inference is applied to help approximate the intractable posterior and to achieve a disentangled representation in the latent space dedicated to the interested source , guided by the prior knowledge applied to train sVCD . Essentially , while y is available for supervised training , sVCD takes x as the sole input during the testing phase to obtain ŷ . To summarize , our main contributions include : • We prove that nonlinear source extraction can be formulated as modified variational inference and achieved by maximizing a tighter variational lower bound . • We design a novel sequence-to-sequence ( Seq2Seq ) translation architecture ; it leverages the modified variational inference to extract the desired source from nonlinear mixture . • We provide empirical evidence that sVCD outperforms a state-of-the-art method based on extensive evaluations under different nonlinear mixing scenarios . The rest of the paper is organized as follows . Section 2 presents related works . Section 3 introduces the mathematical framework for source extraction from nonlinear mixture . Section 4 presents the details sVCD model . Section 5 introduces the datasets and reports the evaluation results . Finally , Section 6 concludes this paper and points out potential future applications . 2 RELATED WORKS . Numerous algorithms have been developed to solve the BSS problem ( Comon & Jutten , 2010 ) : they all aim to decompose signal mixtures into individual components without knowing information about the source signals or the mixing process . Among them , Non-negative Matrix Factorization ( NMF ) decomposes the signal by using low-rank approximations under the non-negative constraint ( Wang & Zhang , 2012 ) . Independent Component Analysis ( ICA ) assumes that the underlying signals are statistically independent to separate them ( Hyvärinen & Oja , 2000 ) . BSS achieved by masking and re-weighting the frequency spectrograms have also been implemented by Hidden Markov Models ( HMMs ) ( Roweis , 2000 ) and segmentation methods ( Bach & Jordan , 2005 ) . One common assumption of regular BSS is that the mixing process is linear , under which the statistical independence of the underlying sources is a sufficient condition to constrain the linear unmixing function . However , as we extend BSS to nonlinear mixture , the statistical independence of the sources can no longer be treated as a sufficient constraint for the unmixing ( Hyvärinen & Pajunen , 1999 ) . Harmeling et al . ( 2003 ) and Hyvärinen & Morioka ( 2016 ) have proposed to utilize temporal structure in the mixture to overcome this challenge . Furthermore , several properties on the mixed sources , e.g. , autocorrelation ( Sprekeler et al. , 2014 ) , general non-Gaussian temporal dependency ( Hyvärinen & Morioka , 2017 ) , or non-stationarity ( Hyvärinen & Morioka , 2016 ) , can also provide sufficient constraints for solving the problem . Aother broadly related topic is disentanglement representation learning ( Bengio et al. , 2013 ) . The word “ disentanglement ” means decoupling of generating factors . Unlike BSS performing separation in signal space , disentanglement representation learning uncovers the underlying factors of variation in latent space . With the development of deep learning , disentanglement representation learning based on VAEs has gained momentum ( Kingma & Welling , 2013 ) . By augmenting the lower bound formulation with the coefficient that regulates the independence prior , Higgins et al . ( 2016 ) proposes β-VAE , a framework to discover interpretable latent representations automatically . Subsequent works ( Burgess et al. , 2018 ; Rolinek et al. , 2019 ; Chen et al. , 2018 ) have led to significant accomplishments in understanding the capabilities of nonlinear disentanglement in VAEs . Researchers have also been seeking the formal definition of disentanglement in several works ( Ridgeway , 2016 ; Higgins et al. , 2018 ; Eastwood & Williams , 2018 ) . The aforementioned algorithms assume that the sources are completely unknown , however , in many practical cases , we have some prior knowledge available and do not actually have to perform source separation in a completely blind manner . As such , these algorithms can be extended to incorporate the knowledge from the priors . In Calhoun et al . ( 2005 ) , paradigm information was employed to aid ICA analysis , and the results are shown to be more robust to noise and outperform basic ICA . In another work ( Jeong et al. , 2009 ) , the original NMF algorithm for BSS is significantly improved by enforcing a constraint that requires disjointness under the semi-blind denoising framework . Last but not least , Kameoka et al . ( 2018 ) achieves source separation by leveraging conditional VAE ( Sohn et al. , 2015 ) with source class as labels in priors during training . All the aforementioned approaches do not seem to suit our need for extracting specific source ( s ) . 3 VARIATIONAL INFERENCE FOR SIGNAL EXTRACTION . Unlike VAE ( Kingma & Welling , 2013 ) that aims to recover input by finding an efficient encoding in latent space , our sVCD focuses on extracting a specific component from nonlinear mixture . Therefore , if we directly borrow the idea of variational inference from VAE , the variational lower bound will not be as tight . As such , we propose and prove a new lower bound for our problem setup , so as to endow a new mathematical framework for sVCD . Let x be a nonlinear mixture of several independent sources . In order to find a representation of x , Bayesian modeling ( Lee , 1989 ) can be used to encode the beliefs about the processes that generate x from all components into a model M with latent vectors z and parameters θM . The model is then “ learned ” by inferring z and adjusting the parameters θM . In other words , given the observed mixture x , Bayesian modeling aims to infer the posterior p ( z|x ) . However , since the posterior distribution is intractable , an optimization approach , which is called variational inference ( Blei et al. , 2017 ) , can be leveraged to approach the problem . To be specific , a surrogate distribution q ( z ) is employed to approximate p ( z|x ) by minimizing their KL divergence ( Van Erven & Harremos , 2014 ) . However , the KL divergence itself still involves the intractable posterior p ( z|x ) . To cope with this issue , the KL divergence is decomposed into : KL ( q ( z ) ‖p ( z|x ) ) = log p ( x ) − Eq ( z ) [ log p ( x , z ) q ( z ) ] . ( 1 ) Since the marginal log-likelihood log p ( x ) is independent of the variational distribution q ( z ) , the KL divergence can be minimized by maximizing the variational lower bound Eq ( z ) [ log p ( x , z ) q ( z ) ] . As a particular type of variational inference method , VAE uses deep neural networks to approximate the generative models . It should be noted that in VAE , the distribution q is conditioned on the observation x , and approximated by an encoder network q ( z|x ) . As such , the variational lower bound of VAE can be expressed as : Eq ( z|x ) [ log p ( x , z ) q ( z|x ) ] = Eq ( z|x ) [ log p ( x|z ) ] −KL ( q ( z|x ) ‖p ( z ) ) , ( 2 ) By slight modifications ( Higgins et al. , 2016 ; Kim & Mnih , 2018 ) , the latent representation z of VAE shows disentanglement properties , suggesting that a regular VAE can be used for separating mixed sources . However , unlike the scenario faced by VAE which is unsupervised and source-agnostic , our task is not completely “ blind ” , i.e. , prior knowledge about a specifically interested component y is available . In this case , the variational lower bound should be Eq̃ ( z|y ) [ log p ( y , z ) q̃ ( z|y ) ] , where q̃ ( z|y ) is an approximate posterior distribution in the form q̃ ( z|y ) = ∫ p ( z|x ) q ( x|y ) dx . It can be observed that the posterior q̃ ( z|y ) takes into account the effect of the mixing process q ( x|y ) , thus covering a broader class of distributions and becoming more expressive . We also notice that there is still the desired component y in the denominator . Considering the goal of extraction signal component from a nonlinear mixture , we would like to maximize Eq̃ ( z|y ) [ log p ( y , z ) q ( z|x ) ] instead , in which case x and y should be used as the input and output of the network , respectively . Therefore , we need to study this new lower bound specifically . Lemma 3.1 . We get a tighter lower bound for the source extraction problem : log p ( y ) ≥ Eq̃ ( z|y ) [ log p ( y , z ) q ( z|x ) ] ≥ Eq̃ ( z|y ) [ log p ( y , z ) q̃ ( z|y ) ] . ( 3 ) Proof . For the left inequality , by using Jensen ’ s inequality , we have : Eq̃ ( z|y ) [ log p ( y , z ) q ( z|x ) ] = ∫ q̃ ( z|y ) log p ( y , z ) q ( z|x ) dz = Eq ( x|y ) Eq ( z|x ) [ log p ( y , z ) q ( z|x ) ] ≤ logEq ( x|y ) Eq ( z|x ) [ p ( y , z ) q ( z|x ) ] = log p ( y ) . ( 4 ) For the right inequality , by using Jensen ’ s inequality , we have : Eq̃ ( z|y ) [ log p ( y , z ) q ( z|x ) ] = Eq̃ ( z|y ) [ log p ( y , z ) ] − Eq̃ ( z|y ) [ log q ( z|x ) ] ≥ Eq̃ ( z|y ) [ log p ( y , z ) ] − Eq̃ ( z|y ) [ q̃ ( z|y ) ] = Eq̃ ( z|y ) [ log p ( y , z ) q̃ ( z|y ) ] . ( 5 ) As a result , we can optimize the surrogate lower bound Eq̃ ( z|y ) [ log p ( y , z ) q ( z|x ) ] of sVCD , which is tighter than that of VAE , for the task of source extraction . This variational inference can be readily implemented by the sVCD network in Section 4 .

Summary: The paper proposes an approach for supervised non-linear regression for multivariate time-series using a sequence-to-sequence approach with self-attention and generative prior on the latent codes. The paper poses this as a source extraction from a nonlinear mixture.  The paper shows how this can be applied to a synthetically created data set as well as two real world data sets.  The first is heartbeat and respiration from radio frequency.  The second is  EEG with  EOG and the goal is to remove the EOG from the EEG. The paper shows that the model is able to extract the signals in the example cases.