Variational Component Decoder for Source Extraction from Nonlinear Mixture
Authors: Shujie Zhang
Tianyue Zheng
Zhe Chen
Jun Luo
Sinno Pan
Venue: ICLR 2022 Submitted (2022)
Keywords: Source Extraction
Variational Inference
Disentanglement

Abstract: In many practical scenarios of signal extraction from a nonlinear mixture, only one (signal) source is intended to be extracted. However, modern methods involving Blind Source Separation are inefficient for this task since they are designed to recover all sources in the mixture. In this paper, we propose supervised Variational Component Decoder (sVCD) as a method dedicated to extracting a single source from nonlinear mixture. sVCD leverages the sequence-to-sequence (Seq2Seq) translation ability of a specially designed neural network to approximate a nonlinear inverse of the mixture process, assisted by priors of the interested source. In order to maintain the robustness in the face of real-life samples, sVCD combines Seq2Seq with variational inference to form a deep generative model, and it is trained by optimizing a variant of variational bound on the data likelihood concerning only the interested source. We demonstrate that sVCD has superior performance on nonlinear source extraction over a state-of-the-art method on diverse datasets, including artificially generated sequences, radio frequency (RF) sensing data, and electroencephalogram (EEG) results.

Under review as a conference paper at ICLR 2022
VARIATIONAL COMPONENT DECODER FOR SOURCE EXTRACTION FROM NONLINEAR MIXTURE
Anonymous authors Paper under double-blind review
ABSTRACT
In many practical scenarios of signal extraction from a nonlinear mixture, only one (signal) source is intended to be extracted. However, modern methods involv- ing Blind Source Separation are inefficient for this task since they are designed recover all sources in the mixture. In this paper, we propose supervised Variational Component Decoder (sVCD) as a method dedicated to extracting a single source from nonlinear mixture. sVCD leverages the Sequence-to-sequees translation ability of specially designed neural network to approximate a non- linear inverse of the mixture process, assisted by priors of the interested source. In order to maintain the robustness in the face of real-life samples, sVCD com- bines Seq2Seq with variational inference to form deep generative model, and it is trained by optimizing variant of variational bound on the data likelihood concerning only the interested source. We demonstrate that sVCD has superior performance on nonlinear source extraction over a state-of-the-aat method on di- verse datasets, including artificially generated sequences, radio frequency (RF) sensing data, and electroenncephalogram (EEG) results.
(Seq2Seq)
INTRODUCTION
Signal extraction from nonlinear mixture is a recurring yet hard research problem in signal pro- cessing and representation learning. To tackle this problem, conventional methods be divided into two categories. One is Blind Source Separation (BSS) that recovers all sources in the mix- ture (Comon & Jutten 2010). the other is leverage available prior knowledge to extract only the desired components from the mixture (Leong et al. 2008). Following the hardness results of nonlin- ear BSS based on nonlinear Independent Component Analysis (ICA) (Hyvärinen & Pajunen 1999). many attempts have been made to realize nonlinear ICA and to further connect it with representation learning (Hyvärinen & Morioka 2016 2017; Khemakhem et al.] 2020). However, in many practi- cal scenarios, only one or few sources supposed to be retrieved for the concerned application: for example, heartbeat monitoring, especially that achieved by contact-free approaches (Ha et al.) 2020). demands the heartbeat signal (waveform) to be extracted out of the nonlinear mixture with other signals such as body movements and respiration. Compared with nonlinear BSS, nonlinear source extraction ing only the source of interest could significantly reduce the computation cost for the same category of algorithms (e.g., deep-learning based signal extraction would be computationally cheaper than deep-lcarning based signal separation). On the other hand, if prior knowledge of the source is avail- able, the training may substantially leverage such knowledge as sufficiently improve the (later) inference accuracy.
be far more efficient and effective under these scenarios: on hand, extract-
Recently, there is a rising interest in investigating disentangled representation learning exploiting deep learning techniques. Different from conventional source separation methods that demand hard- coded rules, a deep learning model aims to learn the underlying generating factors automatically and adaptively. To this end, the frameworks of variational auto-encoder (VAE) (Kingma & Welling, 2013) and generative adversarial network (GAN) (Goodfellow et al., 2014) are widely used, and variants modifying designs and training techniques to enhance disentanglement have also been pro- posed, such as 3-VAE (Higgins et al. 2016). However, as the existing proposals mostly focus on disentangling on the representation in image video data (Kingma & Welling 2013; Higgins al.) 2016 Fabius & Van Amersfoort, 2014) Yingzhen & Mandi 2018) where signal mixture never take place, we cannot directly apply the idea of disentangled representation learning our problem of

---

Under review as a conference paper at ICLR 2022
Target Signal
Extracted
Sources Signals
Signal
w Wolnena
Decoder
Nonlinear Mixture
Pe p()12)
Observed Signals x
Encoder
4p p</xx)
Figure 1: Source extraction from nonlinear mixture with sVCD.
source extraction (a special form disentanglement) from nonlinear time-serics mixture. Though a few results have been obtained for disentangling the sources in nonlinear mixing using deep learning, they fall into the category of BSS (Hyvärinen & Morioka, 2016; Khemakhem et al. 2020).
In this paper, we propose supervised Variational Component Decoder (sVCD) as robust deep- learning framework for source extraction from nonlinear mixture. As illustrated in Figure | s sVCD aims recover the interested source from observed signals x formed by source signals mixed in a nonlinear manner. In particular, sVCD involves an encoder to analyze the temporal features of the observed signals and hence encode them the hidden state (posterior distributions in latent space), and it then decodes the hidden state to recover the interested source. A modified variational inference is applied to help approximate the intractable posterior and to achieve a disentangled representation in the latent space dedicated to the interested source, guided by the prior knowledge applied to train sVCD. Essentially, while y available for supervised training, sVCD takes x as the sole input during the testing phase obtain y. To summarize, our main contributions include:
We prove that nonlinear source extraction can be formulated as modified variational infer- ence and achieved by maximizing tighter variational lower bound.
We design novel sequence- gequence-to-sequee to-sequence (Seq2Seq) translation architecture; it leverages the modified variational inference extract the desired source from nonlinear mixture.
We provide empirical evidence that sVCD outperforms a state-of-the-art method based on extensive evaluations under different nonlinear mixing scenarios.
The of the paper is organized follows. Section 2 the mathematical framework for source extraction from nonlinear mixture. Section 4 presents the details sVCD model. Section [5] introducs the datasets and reports the evaluation results. Finally, Section concludes this paper and points out potential future applications.
presents related works. Section introduces
2 RELATED WORKS
Numerous algorithms have been developed to solve the BSS problem (Comon & Jutten 2010 they all aim decompose signal mixtures into individual components without knowing informa- tion about the source signals or the mixing process. Among them, Non-negative Matrix Factor- ization (NMF) decomposes the signal by using low-rank approximations under the non-negative constraint (Wang & Zhang 2012). Independent Component Analysis (ICA) assumes that the under- lying signals are statistically independent separate them (Hyvärinen & Oja 2000). BSS achieved by masking and re- reiieighting the frequency spectrograms have also been implemented by Hidden Markov Models (HMMs) (Roweis) 2000) and segmentation methods (Bach Jordan 2005).
One common assumption of regular BSS is that the mixing process is linear, under which the statisti- cal independence of the underlying sources is sufficient condition to constrain the linear unmixing function. However, as we extend BSS to nonlinear mixture, the statistical independence of the sources can no longer be treated as sufficient constraint for the unmixing (Hyvärinen & Pajunen, 1999). Harmeling et al. (2003) and [yyvärinen & Morioka (2016) have proposed to utilize tem- poral structure in the mixture to overcome this challenge. Furthermore, several properties on the mixed sources, e.g., autocorrelation (Sprekeler al. 2014), general non- non-Gaussian temporal depen- dency (Hyvärinen & Moriokal 2017), or non non-stationarity (Hyvärinen & Morioka 2016). can also provide sufficient constraints for solving the problem.
2

---

Under review as a conference paper at ICLR 2022
Aother broadly related topic is disentanglement representation learning (Bengio al. 2013). The word "disentanglemenn" means decoupling of generating factors. Unlike BSS performing separation in signal space, disentanglement representation learning uncovers the underlying factors of varia- tion in latent space. With the development of deep learning, disentanglement representation learn- ing based VAFs has gained momentum (Kingma & Welling, 2013). By augmenting the lower bound formulation with the coefficient that regulates the independence prior, Higgins et al.) (2016) proposes V-VAE, a framework discover interpretable latent representations automatically. Subse- quent works (Burgess et al., 2018 Rolinek et al. [219 Chen et al. 2018) have led to significant ac- complishments in understanding the capabilities of nonlinear disentanglement in VAEs. Researchers have also been seeking the formal definition of disentanglement in several works (Ridgeway) 2016
Higgins et al. 2018 Eastwood & Williams 2018).
The aforementioned algorithms assume that the sources are completely unknown, however, in many practical cases, we have some prior knowledge available and do not actually have to perform source separation in completely blind manner. As such, these algorithms can be extended to incorporate the knowledge from the priors. In Calhoun et al. (2055), paradigm information was employed to aid ICA analysis, and the results are shown to be more robust noise and outperform basic ICA. In another work (Jeong et al.] 2009),, the original NMF algorithm for BSS is sigmificantly improved by enforcing constraint that requires disjointness under the semi-blind denoising framework. Last but not least, Kameoka et al.] (2018) achieves source separation by leveraging conditional VAE (Sohn) al. 2015) with source class labels in priors during training. All the aforementioned approaches do not seem to suit our need for extracting specific source(s).
3 VARIATIONAL INFERENCE FOR SIGNAL EXTRACTION
Unlike VAE Kingma& & Welling 2013) that aims to recover input by finding an efficient encoding in latent space, sVCD focuses extracting specific component from nonlinear mixture. There- fore, we directly borrow the idea of variational inference from VAE, the variational lower bound will not be tight. As such, we propose and prove new lower bound for our problem setup, as to endow a new mathematical framework for sVCD.
Let be nonlinear mixture of several independent sources. In order to find representation of x, Bayesian modeling (Lee) 1989) can he used to encode the beliefs about the processes that generate from all components into model M with latent vectors z and parameters 0.- The model is then "learned" by inferring z and adjusting the parameters M. In other words, given the observed mixture x, Bayesian modeling aims to infer the posterior p(z]x). However, since the posterior distribution is intractable, an optimization approach, which is called variational inference (Blei et al.) 2017). can be leveraged to approach the problem. To be specific, surrogate distribution qz employed to approximate p(z[x) by minimizing their KL. divergence (Van Erven & Harremos 2014). However, the KL. divergence itself still involves the intractable posterior p(z]x). To cope with this issue, the KL divergence is decomposed into:
p(x, 2) q(z)
KL (g(z) (p(z(x)) logp(x) Ex(()) log
Since the marginal log- likeliidddood log p(x) is independent of the variational distribution (zz). the KL divergence can be minimized by maximizing the variational lower bound E,(x)) log z(x As a partieular type of variational inference method, VAE uses deep neural networks to approximate the generative models. It should be noted that in VAE, the distribution is conditioned on the observation x, and approximated by an encoder network q(zlx). As such, the variational lower bound of VAE can be expressed as:
((z)
p(x x, z) qz(xx)
(x(())))
log
(xtt))) (log p(x(() KIL (g(z)x) [p(z)),
(2)
By slight modifications (Higgins et al. 2016 Kim & Mnih 2018) , the latent representation of VAE shows disentanglement properties, suggesting that a regular VAE can be used for separating mixed sources. However, unlike the scenario faced by VAE which unsupervised and source-agnostic,
our task is not completely "blind", i.e., prior knowledge about specifically interested component y
is available. In this case, the variational lower bound should be Eq(ely) log [[[[)]
p(y.z) where g(zly) is
3

---

Under review as a conference paper at ICLR 2022
an approximate posterior distribution in the form q(zly) = f (z/x)) (xly) dx. It can be observed that the posterior g(zly) takes into account the effect of the mixing process (xxly), thus covering broader class of distributions and becoming more expressive. We also notice that there is still the desired component in the denominator. Considering the goal of extraction signal component from
a nonlinear mixture, we would like maximize Egj(l)) log ş(žix
p(y..) instead, in which case x and y
should be used as the input and output of the network, respectively, Therefore, we need to study this new lower bound specifically.
Lemma 3.1. We get righter lower bound for the source extraction problem:
ply. z)
p(y, Z) 7(zly)
log p(y) 2 Eg(ayy)
log
2 Bg(x x) log
(3)
Proof. For the left inequality, by using Jensen's inequality, we have:
log p(y, z)
ply, z)
log ply. 2) a (z/xx)
Bx(xly)
= q(zly) log
dz = B,(x v) B((())))
(z/x)
g (z
p(y, 2)
I log Bx( B((x)x)
= ply).
(4)
9 (z)x)
For the right inequality, by using Jensen's inequality, we have:
ply:2 z)
Eg(xly) log
Eg(syy) [log ply, []]] Ba(aly) [log a (z/x)]
q (z x)
Ba(sy [log (ogg(yyz))) kx [0(zly]] = Egaly)) log p(y, z)
(5)
q(zly)
As a result, we can optimize the surrogate lower bound Eg(aly) [log p(y.x) tighter than that of VAE, for the task of source extraction. This variational inference can be readily implemented by the sVCD network in Section
of sVCD, which is
y(
4 SUPERVISED VARIATIONAL COMPONENT DECODER ARCHITECTURE
To implement the variational inference framework for extracting signals from nonlinear mixture as presented in Section 3 we specifically design sVCD network to take observed nonlinear signal mixtures (containing multiple components) as input and to output the temporal sequence of the desired component. It inherits from encoder-decoder architecture with encoder and decoder, both based on Recurrent Neural Network (RNN) (Cho et al. 2014), sandwiching a latent space. The architecture of sVCD is shown in Figure 2 and we elaborate the details in the following.
4.1 ENCODER
We employ an RNN-based encoder to implement the g(zlx) of the variational inference as intro- duced in Section 3 The encoder is parameterized by ø. As demonstrated in Figure Ba the RNN- based encoder 90 ( sVCD takes in the temporal signal Φ1.7 = [x],... It. Ør. of length T,, where w, is abbreviated for æ; E RN, NN represents number of observed signal mixtures. The encoder takes the responsibility of decomposing and selecting the desired component, which is in turn encoded to compressed latent distribution z. Samples drawn from z are then used to drive the decoder for recovering the desired component. We use RNN form the encoder as is naturally suitable for sequence modeling. At each timestamp f, the RNN reads the symbol æ; and previous hidden state denoted as et-1 that summarizes the past information, and updates the new hidden state e,. In general, RNNs model sequences by parameterizing a factorization of the joint sequence probability distribution as a product of conditional probabilities such that:
p (x1,X2 XT)
p (xx) 1=1
II
p (xx, g- (c.-1),
(7)
4

---

Under review as a conference paper at ICLR 2022
where 9r is function that maps the RNN hidden state et-1 a probability distribution over possible outputs, and T is the parameter set of g. RNN can be implemented with gated activation functions for longer dependecies, such Long Short- 1997] or Gated Recurrent Unit (GRU) (Cho et al. 2014). In this paper, we apply bidrectional GRU to for sequential learning. Given an input sequence æ1. the bidirectional GRU, we obtain the encoded features E1.7 = eTi ;eT,
Memory (LSTM) (Hochreiter & Schmidhuber
We let the prior distribution zt ( X() be Gaus- sian with a diagonal covariance matrix, whose mean and log- variance are parameterized by neural networks (usually imple- mented by fully connected layer), and the values depend on deterministic encoded features e;:
Input signal X
Linear
Linear
Linear
Nr W + b", log Σ, = W +b.
(8)
4u.
k1:7
Pi.
Then Ze is sampled from mean and variance by reparameter- ization tricks (Kingma & Welling 2013). We denote the en- coder parameters for computing the input features as Φr and the parameters for computing the mean and variance de, can model encoder as:
A ttention
a1T
T
Encoder GRU
Lol (o(ZET/AZTT)
II 90. xx)) Il g0. (zelet).
(9)
c1r
As such, the encoder has transformed the input temporal se- quence to distributions on the latent space, which can be then readily sampled to drive the decoder for recovering the desired component y.
Linear
Linear
log x²
Sample
Z1:7
4.2 ATTENTION MECHANISM
Linear
Although RNN introduced in Section 4.1 is good way to handle sequential data, they may still lack the ability to dis- cover very long dependencies across thousands of steps. To solve this problem, employ the scaled dot product attention mechanism (Vaswani et al., 2017) to add more contexts to the temporal sequence; the modified encoder architecture is shown in Fig. 35) The core idea of attention is to focus on the most relevant parts of the input sequence for each output. An atten- tion function can be described as mapping a query and a set of key-value pairs to an output. Specifically, the query qı.r, key ki.r, and value Ui.r can be deemed as modified versions of the input æ1T by three different linear transformations as follows:
Decoder GRU
du
Linear
Output signal yı.
Figure 2: Architecture of sVCD.
91:T W + ba
(10)
k1.T W 21. + b.:
(11)
V1.T W 21"
(12)
where W and b (with respective superscripts) are parameters that are trained transform the input to its corresponding query qı:r, key k1-r, and value V1.r, whose dimensions are denoted by dg
dy. d,. The output context G1:T obtained by calculating a weighted sum of the values, where the weight of each value is a normalized product of the query and its corresponding key. Then the context is concatenated to the query to form the final output of our attention layer. In summary, the attention mechanism can be expressed as follows:
(22 qu Q...kkif KTT VL.T
C1.T = softmax
(13)
dd
01.T = (CI. @ q1:T),
(14)
where represents concatenation. Compared with its input Ti.r. the output of the attention layer C1r7 contains more contexts of the whole signal by taking into account the interactions among the elements, thus forming better input for the ensuing RNN
5

---

Under review as a conference paper at ICLR 2022
-
Zt
2x+1
Zt-1
Zt
2++1
--1
Zt
Z++1
c--
Cc
C++1
h
hx
1t++1
--1
Ct
at-1
a2
ac+1
d
d2
de+1
xt-1
X(+1
Xt-1
X
X(+1
Yt-1
Y'
V(+1
(a) Encoder.
(b) Encoder with self-attention.
(c) Decoder.
Figure 3: Architecture of the encoder and decoder.
4.3 DECODER
To realize pe (z)(z) the variational inference problem discussed in Section 3 we implement decoder using RNN-based network, shown in Figure [d] The decoder (3 (.r) is param- eterized by 0, and serves to reconstruct the source of interest yı.T from a latent vector 21:7. The decoding consists of three steps. Firstly the latent vector 21.7 is expanded to hidden feature di- mension by a fully connected layer:
h W 24 +b
(15)
with Wh and bR being the weight matrix and bias represented by the linear layer. Then h is sent the sequence decoding layer implemented by forward directional GRU that produces output d.
Finally, we have the decoded output sequence d1.r = [d. ... di dr). where feature at each step connected two linear layers with rectified linear units (ReLU) to the dimension of yı.r-
yt
W (ReLU de + by bad
(16)
where W81 Wa2 bdd and baz the weights and biases of the linear layer. Overall we have the decoder represented as:
T
T
Po(UIr|ZEET) II Po. (ha 2.) II po, (d, he))
II t=1
po, (v))d))
(17)
4.4 LOSS FUNCTIONS
As stated in Section maximize the variational lower bound Ea(2 y) log p(y22) 2)
in Equation 33
to solve the variational inference problem. Since we have parameterized the probabilities with deep neural networks, we rewrite the variational lower bound as (gqilly) composed as:
In (o(x))
and it can de-
qo (2/x)
Eq(xly) In po 2) Us (z|x))
In Po (y, 2)
(xly) Bgg (2/x)
2s (z/x))
(x)o)))
[I Po (y [z)]
KL (gs (z|x) [zo(
(18)
This lower bound can be separated into two parts, the first term is a reconstruction error between the sVCD output and the ground truth, and the second term as regularizer which enforces the approximate posterior to be close to the specified prior. To practically implement the reconstruction loss (x)))) [In po (y (z)], we use the L2 distance between the separated component ý and y:
LRC 11 y
(19)

---

Under review as a conference paper at ICLR 2022
Since the prior distribution is defined to be unit Gaussian, the second term can be practically imple- mented by:
LRE = KL (N (m,, Σ2) NNO,,,
(20)
And the loss function of vCCD can be defined as:
LVCD = LRC + XCRE.
(21)
where À is the weight controlling the strength of the regularization.
5 EVALUATION
In this section, we first report experimental results on three datasets for sVCD compared with state- of-the-art baselines. We then perform an ablation study on sVCD by removing its certain parts, in order to gaining more understanding of the network.
5.1 DATASETS FOR NONLINEAR MIXTURES
Traditional datasets for BSS, e.g., those created for audio source separation (Garofolo et al. 1993 Vincent et al. 2007) are mostly linear. To demonstrate the nonlinear mixture separation efficacy of sVCD, we employ three datasets of nonlinear mixtures, with details elaborated in the following.
Artificially Generated Dataset One artificially generated dataset is employed in our evaluation due to its special statistical properties. For this dataset, both source signals and observed signals are generated according to the settings in (Hyvärinen & Morioka 2017). All source signals are randomly produced from an AutoRegressive (AR) model with non non- Gaussian process. Then, we obtain nonlinear mixtures from the source signals leveraging nonlinear mixing function comprised of multi-layer perceptron (MLP) with nonlinear activation functions, i.e., leaky ReLU units. For each of the 4,000 trials, we leverage the model to generate 12 temporally dependent sources (inde- pendent of each other) that are in turn mixed to produce 12 observed signals; we select one as source of interest be extracted by sVCD.
RF-PPG Dataset Contact-free vital sign sensing the problem of extracting vital signs of human subjects via contact-free sensors, a representative and practical scenario for signal extraction from the nonlinear mixture. To be specific, the micromovements of the skin caused by heartbeats are mixed non- non-linearly with respiration- iuspirationinndueed induced movements and other body movements, and the mix- ture is carried on RF signals reflected back from human bodics. In this dataset, a high-resolution radar sensor (Novelda ASI (017) is used to collect the RF signals. The ground truth heartbeat hhatopletyssmoraphyy (PPG) waveforms are measured by NeuLog (2017). The data collection is conducted with 12 healthy subjects (6 females and 6 males) under the IRB approval of our institute. The subjects are asked to sit at quasi-static and natural states. The RF-PPG dataset includes 48-hour data, approximately 180,000 heart cycles in total, and is divided into 20-seconds segments.
EEG-EOG Dataset Electroencephalograppy (EEG) is electrophysiological monitoring tech- nique for recording electrical activity on the scalp, which has been proven to reflect the activity of the brain's surface layer underneath (Henry) 2006). Due the sophisticated structure of the brain, the signals at different locations of the underneath brain (whose ground truths cannot be obtained directly), when manifested as EEG signals, are often superimposed nonlinearly and exhibit complex behavior (Steyrl et al. 2013). and the nonlinearity is escalated by the nonlinear properties of the brain (Nunez et al.) (006)) and the skull (Zhang et al. 2014). Moreover, the electrical activity of the eye, which is the main source of artifacts of brain signals, complicates the EEG recordings. As such, it often required to isolate and remove electrical signals of the eyes, i.c., electrooculography (EOG) signals, from EEG recordings. In the collected dataset (Torkamani-Azar et al.) 2020), EEG and EOG signals are recorded simultaneously to form 2.5 2.5-minute segments. The EEG signals have 64 channels corresponding to electrodes mounted on the scalp, and the EOG signals have channels corresponding to electrodes connected to the outer corners of both left and right eyes, and the middle of the eyebrows. The dataset is collected from 10 human subjects for duration of 105 minutes.

---

Under review as a conference paper at ICLR 2022
5.2 EXPERIMENT SETUP
We employ the aforementioned datasets to evaluate the performance of sVCD. The collected datasets divided into 80% training and 20% test sets. For the model parameters, we set the size for d,. dg and dy as 64; e, 2. h. d are set respectively as: 64, 8, 128, 128; and dropout rate at 0.2. The weight is set to 0.1 by line search. For the training process, we employ the Adam optimizer (Kingma & Ba 2014) with starting learning rate of 0.001 to update the parameters according to equation 21 The batch size is
20, To enable proper evaluation, all signals are scaled to [-1, 1, 1] beforehand.
To evaluate the performance of the method we calculate the cosine similarity between the recovered component y and the ground truth component y. The cosine similarity defined as S =
y.y Iy = VET= 82 bi with value the range of [-1, 1]. A high similarity indicates that better performance of the model, and vice versa.
Ens Seat
To compare sVCD with previous works, choose ICA (Hyvärinen & Oja 2000) and Conv- TasNet (Luo & Mesgarani 2019) as baseline methods; they are both designed for separating linear widely used blind source separation technique, and the fitting of ICA and the training of sVCD are similar since both consider the context of the signal including both the past and the future. Conv-TasNet chosen the baseline because it is state- of-the-art deep-learning method for automatic signal separation. We follow the parameter settings in the original papers, and for ICA, we separate 12, 5, and 64 independent components from the artificially generated, RF-PPG, and BEG-EOG datasets, respectively.
signal mixture. ICA chosen because
5.3 RESULTS
In Figure 4 we show the waveforms of the ground truth, and the signals extracted by ICA, Conv- TasNet (abbreviated as C-TasNet) and sVCD signal on the three datasets. For artificially gen- erated data, sVCD retrieves component approximately the same the ground truth, while ICA and Conv-TasNee obtain very noisy results. For the RF-PPG data, sVCD successfully recovers the heartbeat waveform bearing correct heart rate, while ICA barely separate heartbeat signal from res- piration as the breath waveform is still dominating, and Conv-TasNe fails to recover event details in the waveform. For the EEG-EOG dataset, sVCD extracts source signal bearing high similarity to the ground truth, while ICA and Conv-TasNet seem to have deduced totally wrong outcomes.
Amplitude
Amplitude
Amplitude
TusNet
10
15
Timeis)
Time(s)
Time(s)
(a) Artificially generated data.
(b) RF-PPG data.
(c) EEG-EOG data.
Figure 4: Source signals extracted by ICA, Conv-TasNet, and sVCD, against the ground truth (GT).
Cosine Similarity
Cosine Similarty
Cosine Similarity
0.5
0.5
0.5
C-TasNet sVCD (a) Artificially generated data.
ICA
ICA
C-TusNet sVCD
ICA C-TasNet sVCD
(b) RF-PPG data.
(c) EEG-EOG data.
Figure 5: Statistics cosine similarities for respective datasets.
8

---

Under review as a conference paper at ICLR 2022
In Figure 5 show the cosine similarities between the recovered waveforms and the ground truth of ICA, Conv-TasNet, and sVCD. It can be observed that the performance of sVCD substantially surpasses those of ICA and Conv-TasNet on all three datasets, and the median cosine similarities of sVCD are all above 0.8 on all three datasets, indicating a strong resemblance between the s sVCD) extracted source signals and the ground truth.
5.4 ABLATION STUDIES
We provide ablation study of the main design decisions for sVCD in this Section. In the first study labeled as "Seq2Seq". we skip the variational inference part, and make the output of the encoder GRU directly driving the decoder GRU. Meanwhile, in the second study "sVCD with only mean regularization", instead of regularizing the distributions to the standard Gaussian, we regularize only the mean value towards O. Lastly, the study labeled as "sVCD without attention" holds out the attention mechanism. Our goal is to see the respective accuracy drops to measure their importance in the proposed sVCD network. Given the cost of training a single model, we did not compute confidence intervals for each variation. The result of the ablation study is shown in Table
We can observe that our proposed sVCD achieves cosine similarities of 0.9358, 0.8634, and 0.7803 on the three datasets, respectively. Performance drops can be observed for the Seq2Seq model on Artificially generated, RF-PPG and EEG-EOG datasets, indicating that a discrete latent space without regularization encodes less useful information that can be used by the decoder to recover the source. For the sVCD with only mean regularization, further performance decrease can be observed, proving that the distribution mean and variance should better off be regularized together. As for the sVCD without attention, some clear drop cosine similarity can observed on all three datasets, demonstrating that the attention mechanism helps boosting the performance of sVCD.
Table 1:
Average cosine similarities in the ablation studies.
Artificially generated
RF-PPG
EEG-EOG
Proposed sVCD
0.9358
0,8634
0.7803
Seq2Seq
0.9339
0.8593
0.7765
sVCD with only mean regularization
0.8737
0.8499
0.7596
vCC without attention
0.8552
0.8342
0.7577
6 CONCLUSION
Taking an important step toward deep learning enabled signal processing, we proposed new ap- proach for source extraction from nonlinear mixture. We started by formulating nonlinear signal extraction modified variational inference problem. By detailed derivation, we have proved that the mathematical framework of signal extraction can implemented by an sVCD network, which the modified variational lower bound is maximized using backpropagation. Experiments artifi- cially generated data, RF-PPG data, and EEG-EOG data have proved the signal extraction efficacy of vehicle to validate the overall mathematical framework,
sVCD. The sVCD in this paper is used but better architectures are likely to be found. In the future, we plan to apply the sVCD network to other promising applications such source extraction from video data, econometric data, and biomedical data (such as EMG and ECG), in which source signals have nonlinear mixtures.
ETHICS STATEMENT
All experiments involve human subjects have been approved by IRB of our institute, and the experi- ments have been conducted strictly followed the standard procedures. The RE-PPG data will not be made available to the public to protect the privacy of the subjects.
REPRODUCIBILIIT STATEMENT
In this section we briefly reference the details needed to help reproducibility. For the dataset: we have uploaded the artificially generated dataset in the link in supplementary materials; due to

---

Under review as a conference paper at ICLR 2022
privacy concerns could not make the RF-PPG dataset to public; the EEG-EOG dataset is public dataset acquired from Torkamani- et al. 2020). The model implementation. settings are described as in Section 5.2 We have also uploaded the source code to allow for reproducing results in supplementary materials.
REFERENCES
Francis R. Bach and Michael I. Jordan. Blind One-Microphone Speech Separation: A Spectral Learning Approach. Proc. of NIPS, 17:65-72, 2005.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-1888, 2013.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational Inference: A Review for Statis- ticians. Journal of the American statistical Association, 112(518):59--87,, 2017.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding Disentangling in 3-VAE. arXiv preprint arXiv:1804.0333 08499034493 2018.
Vince D. Calhoun, Tülay Adali, Michael C. Stevens, Kent A. Kiehl, and James J. Pekar. Semi- blind ICA of fMRI: A Method for Utilizing Hypothesis-derived Time Courses in Spatial ICA Analysis. Neuroimage, 25(2)5527-538, 2005.
Ricky TQ Chen, Xuechen Li, Roger Grosse. and David Duvenaud. Isolating Sources of Disentan- glement in Variational Autoencoders. arXiv preprint arXiv: 1802. 04942, 2018.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder- Decoder for Statistical Machine Translation. In Proc. of EMNLP, pp. 1724-1734, 2014.
Pierre Comon and Christian Jutten. Handbook of Blind Source Separation: Independent Component Analysis and Applications. Academic Press, 2010.
Cian Eastwood and Christopher K.I. Williams. A Framework for the Quantitative Evaluation of Disentangled Representations. In Proc. of ICLR, 2018.
Otto Fabius and Joost Van Amersfoort. Variational recurrent auto-encoders. arXiv preprint
arXiv: 1412,584,,,,58,,, 2014.
John Garofolo, Lori Lamel, William M. Fisher, Jonathan G. Fiscus, David Pallett, Nancy L. Dahlgren, and Victor Zue. TIMIT Acoustic-Phonetic Continuous Speech Corpus. Web Download, 1993. Philadelphia: Linguistic Data Consortium.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. Proc. of the 27th NIPS, pp. 2672-2680, 2014.
Unsoo Ha. Salah Assana, and Fadel Adib. Contactless Seismocardiography via Deep Learning Radars. In Proc. of the 26th ACM MobiCom, 2020.
Stefan Harmeling, Andreas Ziehe, Motoaki Kawanabe, and Klaus-Robert Müller. Kernel-based Nonlinear Blind Source Separation. Neural Computation, 1085-1109- 2003.
J. Craig Henry. Ellettoennephaaloggapy:: Basic Principles, Clinical Applications, and Related Fields. Neurology, 67 67:11):29222299 2006.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning Basic Visual Concepts with Constrained Variational Framework. In Proc. of ICLR, 2016.
10

---

Under review as a conference paper at ICLR 2022
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a Definition of Disentangled Representations. arXiv preprint 2018.
arXiv:1812.022300, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-term Memory. Neural Computation, 9(8):
1735-1780, 1997.
Aapo Hyvärinen and Hiroshi Morioka. Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gamett (eds.), Proc. of NIPS, pp. 29:1 2016.
Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ICA of Temporally Dependent Stationary Sources. In Artificial Intelligence and Statistics, pp. 460 469.. PMLR, 2017.
Aapo Hyvärinen and Erkki Oja. Independent Component Analysis: Algorithms and Applications.
Neural Networks, (34-5)):41 11340430,,, 2000.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear Independent Component Analysis: Existence and Uniqueness Results. Neural Networks, 12(3):299-439 1999.
So- Young Jeong, Kyuhong Kim, Jae-Hoon Jeong, and Kwang-Cheol Cheoll Oh. Semi-blind Disjoint Non- negative Matrix Factorization for Extracting Target Source from Single Channel Noisy Mixture. In 2009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pp. 73-76. IEEE, 2009.
Hirokazu Kameoka, Li. Shota Inoue, and Shoji Makino. Semi-blin Source Separation with Multichannel Variational Autoencoder. arXiv preprint net.:1888.0092 2018.
llyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvärinen. Variational Autoen- coders and Nonlinear ICA: A Unifying Framework. In Proc. of the 20th AISTATS, pp. 2207-2217. PMLR, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by Factorising. In Proc. of ICML, pp. PMLR, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6988. 2014.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In Proc. of ICLR, pp. 1-14, 2013.
Peter M. Lee. Bayesian Statistics. Oxford University Press London, 1989.
Wai Yie Leong, Wei Liu, and Danilo P. Mandic. Blind Source Extraction: Standard Approaches and Extensions to Noisy and Post-Nonlinear Mixing. Neurocomputingg 71(10-12)2344-2355 2008.
Yi Luo and Nima Mesgarani. Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation. IEEE/ACM transactions audio, speech, and language processing, 27
12): 2019.
NeuLog. Heart Rate & Pulse Logger Sensor NUL-208.
com/
heart rate pulse// 2017. Accessed: 2021-05-30.
Novelda AS. The World Leader in Ultra Wideband (UWB) Sensing. httts::/noveldaa
technology/] 2017. Accessed: 2021-05-30.
Paul L. Nunez, Ramesh Srinivasan, et al. Electric Fields of the Brain: the Neurophysics of EEG. Oxford University Press, USA, 2006.
Karl Ridgeway. A Survey of Inductive Biases for Factorial Representaaiion-LLaminng arXiv preprint arXiv: 161.:296/..05299, 2016.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational Autoencoders Pursue PCA Di- rections (by Accident). In Proc. of the IEEF/CVF CVPR, pp. 12406-12415, 2019.

---

Under review as a conference paper at ICLR 2022
Sam T. Roweis. One Microphone Source Separation. In Proc. of NIPS, volume 13, pp. 2000:1-7, 2000.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning Structured Output Representation using Deep Conditional Generative Models. In Proc. of NIPS, volume 28, pp. 3483-3491, 2015.
Henning Sprekeler, Tiziano Zito, and Laurenz Wiskott. An Extension Slow Feature Analysis for Nonlinear Blind Source Separation. The Journal of Machine Learning Research, 15(1):9219947, 2014.
David Steyrl, Reinhold Scherer, and Gernot R. Müller-Putz. Random Forests for Feature Selection in Non-invasive Brain-computer Interfacing. In International Workshop on Human- Computer In- teraction and Knowledge Discovery in Complex, Unstructured, Big Data, pp. 207-216. Springer, 2013.
Mastanch Torkamani-Azar Sumeyra Demir Kanik, Serap Aydin, and Mujdat Cetin. Prediction of Reaction Time and Vigilance Variability from Spatio-spectral Features of Resting-state EEG in Long Sustained Attention Task. IEEE Journal of Biomedical and Health Informatics, 24(9): 2550 2558, 2020.
Tim Van Erven and Peter Harremos. Rényi Divergence and Kullback-Leibler Divergence. IEEE Transactions on Information Theory, 60(7):3797-3820, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Proc. of NIPS, pp. 5998- 6008, 2017.
Emmanuel Vincent, Rémi Gribonval, and Mark D Plumbley. Oracle estimators for the benchmarking of source separation algorithms. Signal Processing. 87(8) 2007.
Yu-Xiong Wang and Yu-Jin Zhang. Nonnegative Matrix Factorization: A Comprehensive Review.
IEEE Transactions Knowledge and Data Engineering, 25(6):1336 2012.
Li Yingzhen and Stephan Mandt. Disentangled Sequential Autoencoder. In Proc. of the 35th PMLR ICML, pp. 5670-5679, 2018
Jinhua Zhang, Jiongjian Wei, Baozeng Wang, Jun Hong, and Jing Wang. Nonlinear EEG Decoding based on Rarticle Filter Model. BioMed Research International, 2014, 2014.
12

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTS ON LENGTH OF SEQUENCE
As sVCD is built on RNN structure, there is no strict requirement on the length of input sequence. We provide an experiment on the adaptability sVCD by training with sequence length of L, then testing with different lengths from 2L to 0.01L. For the three datasets mentioned in Section 5.1 800 time steps which equivalent to: 16 seconds for artificially generated data, 20 seconds RF-PPG data, 4 seconds for EEG-EOG data. The cosine similarities are presented in Figure 6 It can observed that generally tested with sequence lengths from 0.5L to 2L, resulting statistics do not have much variation for all three datasets. For artificially generated dataset, test data length with only 0.01L still has median similarity above 0.9, but much larger error can be observed. For RF-PPG dataset, test data that have length lower than 0.4Z have similarities much lower than those longer. While for EEG-EOG dataset, test data length lower than 0.5L will make the performance drops. Different minimum sequence length of test data may be due different source signals have distinct time-correlation features. Therefore, is better to take a sufficiently long sequence as hard judge the minimum yet sufficient length a priori. Overall. the results show promising future of sVCD that after training with an adequate length, the model could adapt to not only short time observation but also long time monitoring without extra training efforts.
Cosine Similarity
0.95
0.9
0.85
0.8
0.75
0.01L
0.0SL
0.1L 0.25L 0.5L
0.75L
1.5L
2L
(a) Artificially generated data.
Cosine Similarity
0.6
0.4
0.2
0.1L
0.2L
0.3L
0.4L
0.5L
0.75L
1.5L
2L
(b) RF-PPG data.
Cosine Similarity
0.6
0.4
0.2
0.3L
0.4L
0.75L
L
1.5L
(c) EEG-EOG data.
Figure 6: Statistics cosine similarities for respective testing data sequence length.
13

---

Under review as a conference paper at ICLR 2022
B EXPERIMENTS ON LINEAR AND NONLINEAR REGRESSIONS
We have performed linear and nonlinear regression for the signal extraction purpose. For nonlinear regression, employ least square regression and support vector regression. For each category of nonlinear regression, we employ polynomial, radial basis function and sigmoid kernel. For non- linear regression, we only report the results of support vector regression with polynomial kernel because this combination delivers the best performance. The recovered waveforms of linear regres- sion (denoted "LinReg"), nonlinear regression (denoted as "NLinReg"), and sVCD are shown in Figure 7 and their respective cosine similarities are shown in Figure It can be observed that sVCD significantly outperforms these two regression methods, while both regressions fail to recover meaningful results. The reason for their inferior performance can be readily explained by their omis- sion of temporal correlations, but they do offer instantancous point-to-point conversion instead of the seq-to-seq manner operated by sVCD.
Amplitud
Amplitude
wwmmmmm An
Amplitudo
NLinRes
SVDD
5
10
15
Time(s)
Time( (S)
Time(s)
(a) Artificially generated data.
(b) RF-PPG data.
(c) BEG-EOG data.
Figure 7: Source signals extracted by linear regression, nonlinear regression, and sVCD, against the
ground truth (GT).
Cosine Similarity
Cosine Similarity
Cosine Similarity
0.5
0.5
0.5
-0.5
-0.5
-0.5
sVCD
LinReg NLinReg
sVDD
LinRog NLinReg
sVCD
LinReg NLinReg
(a) Artificially generated data.
(b) RE-PPG data.
(c) EEG-EOG data.
Figure Statisties on cosine similarities for respective datasets.
14


Reviews:
[{"review_id": "no9JESUJDWX", "reviewer": "Reviewer_LaJi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There is no obvious error I can see in the theory or experiments of this paper. Although I was not able to check the math carefully. The results seem sufficient to back the claims made by the authors. Although the techniques themselves are not novel, the application of these techniques to solve the non-linear Blind Source Separation problem is somewhat novel. Blind Source Separation is a ubiquitous problem and even though this work concerns itself just with one restricted version of the problem, where only one source is extracted, the applicability of this work is potentially quite significant. The results presented by the authors are satisfactory and show convincingly that the presented method is superior to other state-of-the-art techniques. I think the paper is good for acceptance, but I think it would be interesting if the authors could also relate their work with Sparse Coding and not just non-linear ICA.  ", "summary_of_the_paper": "Authors present a novel approach that uses deep learning to solve non-linear Blind Source Separation problems. More specifically, their approach combines Seq2Seq and variational inference to extract one source of interest out of the nonlinear mixture. The generative model incorporates the prior beliefs about the source to be extracted.  ", "summary_of_the_review": "The paper introduces a novel application of Seq2Seq and variational inference and the results show that the method's performance is superior to other state-of-the-art methods for a restricted version of Blind Source Separation where only one source is extracted from a non-linear mixture. My recommendation is to accept this paper for the conference. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WmZIBkZ0BT", "reviewer": "Reviewer_epBn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe modeling problems the approach seeks to solve are interesting. \n\nThe papers review of nonlinear source separation and linear source separation helped set the context. \n\nApproaches for Seq2Seq are likely to have an increasing impact in time-series tasks and demonstration of their potential is likely to have an impact in the machine learning for signal processing community. \n\nThe paper's architecture (combination of self-attention, recurrent network, generative code layer, and decoder) appears to be useful for sequence regression problems.\n\nThe experimental results look promising in their accuracy.  \n\nWeaknesses:\n0. The problem boils down to a regression task. However, there are no baseline comparisons with nonlinear regression models. It is not clear how the ICA baseline, which could extract multiple linearly demixed source signals, is fairly compared. Is the ICA component shown in the figures and the best plots the best matched?  ICA is not only a linear approach but it also instantaneous. It doesn't consider past (or future values). Why couldn't MSE be used instead of cosine similarity? Is the model not able to match the mean or magnitude of the target?\n\n1. Much of the time-series aspects are ignored. \na) The paper's architecture does consider nor impose the causal structure on the decoder. That is the self attention creates a non-causal representation.\nb) The effect of the length of the sequences in the regression performance is not discussed. Are all training sequences the same length? Can they vary in testing? \nc) Point about comparing against instantaneous ICA.\n\n2. The EEG-EOG Dataset introduces an inaccurate description of electrophysiology. The electric potentials are instantaneously mixed. The reference cited for 'superimposed nonlinearly' actual refers to the non-linear dynamics not the combination of multiple sources impinging on the electrodes.  While nonlinearities can appear due to the sensing being nonlinear (amplifiers being saturated), this is not the sense of the description. \n\n3. In the ablation study, it appears the Seq2Seq method performs well without the variation inference. However some gap exists and it isn't clear if this could be lowered by keeping the affine transform from the encoding and still regularizing the difference from mean 0.\n\nMinor points:\npage 1, \"proposal mostly focus\" -> \"proposal mostly focus\"\npage 2, \"sour extraction\"\nFigure 2, where are the time indices in the representation fo the encoder and prior distribution? \npage 3, after equation 1 shouldn't the phrasing be that the KL divergence can be minimized by maximizing the variational lower bound?\npage 5, the subscript notation of $1:T$ is not introduced and would seem to be a matrix. Generally the notation follows MATLAB/Octave but it is not clear to the reader.  For instance in equation 13, I am guessing the softmax operates on elements of the outer product by normalizing across each row.  The concatenation notation used in 14 should be defined. \npage 5, first line \"To be realized\npage 5, \"forward bi-directional\" ? \npage 5, $L^2$ distance is actually the squared $L_2$ distance between the signals (or $\\ell_2$ in $\\mathbb{R}^T$)", "summary_of_the_paper": "The paper proposes an approach for supervised non-linear regression for multivariate time-series using a sequence-to-sequence approach with self-attention and generative prior on the latent codes. The paper poses this as a source extraction from a nonlinear mixture.\n\nThe paper shows how this can be applied to a synthetically created data set as well as two real world data sets.  The first is heartbeat and respiration from radio frequency.  The second is  EEG with  EOG and the goal is to remove the EOG from the EEG. The paper shows that the model is able to extract the signals in the example cases.", "summary_of_the_review": "While the paper presents a meaningful architecture for the task, key aspects of the nature of time-series are ignored and no meaningful baselines are used for benchmarking. Thus the paper is preliminary and needs a more thorough set of comparisons against other non-linear regression problems with and without causal restrictions on the modeling. \n\n-----------------------------\n\nThe authors have performed additional comparisons that help set the context. I am willing to raise my score if the text is further clarified to include the fact that is this is a non-linear multiple input and single output system identification model. \n\nI agree the concerns with non-causality may not be valid in all settings, and the additional testing on performance at shorter time segments is important.\n\nFinally, I would like to encourage the revision to have careful wording regrading the non-linearity of the mixing versus non-linearity of processes. These two can get conflated. I agree that sometimes electrical recordings of linearly mixed sources result in nonlinear mixtures due to non-stationaries due to sensor/electrode movement as well as amplifier/recording nonlinearities. \n\nFurther empricial analysis into the performance under challenging non-linearities such as clipping would be encouraged. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ML7wedKCQWY", "reviewer": "Reviewer_MyNd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1] Far above average clarity and overall writing quality and completeness\n2] In section 5.4, the Ablation Study explores limits of the Gaussian assumption\n\nSuggestions\n\n1] \"source\" <-- \"sour\" at the top of page 2\n2] Try comparing against more advanced technique (whther another deep learning model, or an algorithm that has been tuned to the specific application)", "summary_of_the_paper": "Problem: blind source separation of all sources in a nonlinear mixture is very difficult, and potentially wasteful if you only need to extract one component\nSolution: focus on extracting only the component that you need, by training a Variational encoder/decoder architecture (named a variational component decoder) to reconstruct only the component you wish to extract.\n\nAdditional contribution:\n* lower bound of variation for the new architecture is proved", "summary_of_the_review": "The authors make a strong theoretical justification of the model approach to source extraction from time series. The writing is exceptionally clear and informative and the experimental design is exceptionally sound. I scored 3/4  Empirical significance because the method is only compared to ICA, rather than (1) another more comparable deep model, or (2) state-of-the-art method in the specific tasks you are performing on. On the other hand I greatly appreciate that the authors explore the bound of their underlying Gaussian assumption in Sec 5.4. The technical novelty and significance is more exceptional due to the provided bound justifying the approach. Overall I think this is a solid \"accept\", also because the authors did something that worked and can explain why. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JCa59TFzg0q", "reviewer": "Reviewer_aZ4J", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and clear to understand. Proposed method is an elegant extension of a seq-to-seq VAE that exploits an attention mechanism, and reconstructs a source signal (that one needs to have available as a ground truth in the training set -which is a limitation-), based on a variational lower bound on the source signal’s data likelihood. Ablation studies are interesting and nicely performed, as they demonstrate a significant benefit of using an attention mechanism for time-series signal source extraction. In my opinion the paper’s main weakness is in the experimental analyses and evaluations that are presented.\n\nAuthors tried to make a distinction between nonlinear “source separation” and “source extraction” from mixtures. I am not fully convinced on the arguments around this theme. (1) For instance authors present arguments on “significantly reducing the computation cost”, when one is interested only in extracting the source signal of interest (i.e., nonlinear source extraction), rather than performing nonlinear blind source separation. This should be supported later in the experiments (which the authors did not), by discussing computation costs of the proposed algorithm, with respect to e.g., ICA. (2) Another line of argument by the authors state that “existing work on representation disentanglement focuses on image or video data, where signal mixtures never take place”. I do not fully agree on this statement, as in most cases these architectures are also applicable to time-series signal recordings (cf., studies on variational speech separation). To support their claim, authors can maybe present a baseline comparison to a simple VAE-based source separation model.\n\nPerformed experiments on the EEG-EOG dataset do not make much sense from a practical viewpoint, and reveal an important weakness. From what the reader can understand, the ground truth “y” to train the sVCD models for reconstruction is given as the EOG signals that are provided in the dataset. This means that sVCD recovers the best estimate of the EOG time-series from nonlinear EEG mixtures. While it is successfully shown that sVCD can perform this, it is not a meaningful experiment to demonstrate. In nonlinear EEG source separation/extraction, one is always interested in recovering some EEG source signal (e.g., originating from an independent cortical source component), that is clean from artifactual (e.g., EOG, EMG) mixing. I understand that since there is no given ground truth of any EEG source component, it would also be not possible to demonstrate such an experiment with sVCD. However, I do not understand why recovering EOG artifacts from EEG signals would be an interesting demonstration to include in the paper.\n\nOverall, methodological comparisons seem shallow. It is also not clear how did the authors implement ICA (as a comparison baseline) in their experiments? How many independent components were initially extracted and how were the ICs selected following the decomposition? No supplementary or further descriptions of these experiments are presented. For instance, for EEG-EOG experiments, if one decomposes multi-channel EEG into independent components by pooling the training data, it should be generally obvious to recover the EOG component in one of the ICs besides cortical EEG signal source estimates (this is a widely-used approach for EOG artifact removal to discard ICs relevant to eye-blinks [T.-P. Jung et al., NIPS 1997], [S. Makeig et al., PNAS 1997]).\n\nMinor comments:\n- How did the authors determine the KL-term regularization strength? It is also not mentioned what are the parameters used?\n- “sour” -> “source” on page 2, fourth line.\n- “realized” -> “realize” on page 6, first sentence.\n", "summary_of_the_paper": "Authors propose a supervised variational component decoder (sVCD) framework that estimates a single source signal sequence from non-linear mixtures. Proposed model relies on a sequence-to-sequence translating variational encoder-decoder architecture, where optimization is performed based on a variational lower bound on the source signal’s data likelihood. Experiments are performed on artificially generated nonlinear sequence mixtures, RF sensing data and an EEG-EOG dataset.", "summary_of_the_review": "The paper is clearly written and organized. However the paper is rather weak in depicting its contributions with the present experimental analyses and evaluations, and lacks detailed comparisons to other methods. I listed my major concerns in the main review, and would be willing to re-address my rating based on the authors’ responses and revisions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work has generated a lot of discussion between authors and reviewers and among reviewers.
Overall it is reported that the results on EEG are not conclusive and directly relevant for this field.
Besides the theoretical contribution is not reported as a strong point of the work and the
comparison with alternative baseline methods is judged too limited.

For all these reasons the paper cannot be endorsed for publication at ICLR this year.
Decision: Reject