Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors present a novel approach that uses deep learning to solve non-linear Blind Source Separation problems. More specifically, their approach combines Seq2Seq and variational inference to extract one source of interest out of the nonlinear mixture. The generative model incorporates the prior beliefs about the source to be extracted.  
main_review: There is no obvious error I can see in the theory or experiments of this paper. Although I was not able to check the math carefully. The results seem sufficient to back the claims made by the authors. Although the techniques themselves are not novel, the application of these techniques to solve the non-linear Blind Source Separation problem is somewhat novel. Blind Source Separation is a ubiquitous problem and even though this work concerns itself just with one restricted version of the problem, where only one source is extracted, the applicability of this work is potentially quite significant. The results presented by the authors are satisfactory and show convincingly that the presented method is superior to other state-of-the-art techniques. I think the paper is good for acceptance, but I think it would be interesting if the authors could also relate their work with Sparse Coding and not just non-linear ICA.  
summary_of_the_review: The paper introduces a novel application of Seq2Seq and variational inference and the results show that the method's performance is superior to other state-of-the-art methods for a restricted version of Blind Source Separation where only one source is extracted from a non-linear mixture. My recommendation is to accept this paper for the conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Problem: blind source separation of all sources in a nonlinear mixture is very difficult, and potentially wasteful if you only need to extract one component
Solution: focus on extracting only the component that you need, by training a Variational encoder/decoder architecture (named a variational component decoder) to reconstruct only the component you wish to extract.

Additional contribution:
* lower bound of variation for the new architecture is proved
main_review: Strengths
1] Far above average clarity and overall writing quality and completeness
2] In section 5.4, the Ablation Study explores limits of the Gaussian assumption

Suggestions

1] "source" <-- "sour" at the top of page 2
2] Try comparing against more advanced technique (whther another deep learning model, or an algorithm that has been tuned to the specific application)
summary_of_the_review: The authors make a strong theoretical justification of the model approach to source extraction from time series. The writing is exceptionally clear and informative and the experimental design is exceptionally sound. I scored 3/4  Empirical significance because the method is only compared to ICA, rather than (1) another more comparable deep model, or (2) state-of-the-art method in the specific tasks you are performing on. On the other hand I greatly appreciate that the authors explore the bound of their underlying Gaussian assumption in Sec 5.4. The technical novelty and significance is more exceptional due to the provided bound justifying the approach. Overall I think this is a solid "accept", also because the authors did something that worked and can explain why.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach for supervised non-linear regression for multivariate time-series using a sequence-to-sequence approach with self-attention and generative prior on the latent codes. The paper poses this as a source extraction from a nonlinear mixture.

The paper shows how this can be applied to a synthetically created data set as well as two real world data sets.  The first is heartbeat and respiration from radio frequency.  The second is  EEG with  EOG and the goal is to remove the EOG from the EEG. The paper shows that the model is able to extract the signals in the example cases.
main_review: Strengths:
The modeling problems the approach seeks to solve are interesting. 

The papers review of nonlinear source separation and linear source separation helped set the context. 

Approaches for Seq2Seq are likely to have an increasing impact in time-series tasks and demonstration of their potential is likely to have an impact in the machine learning for signal processing community. 

The paper's architecture (combination of self-attention, recurrent network, generative code layer, and decoder) appears to be useful for sequence regression problems.

The experimental results look promising in their accuracy.  

Weaknesses:
0. The problem boils down to a regression task. However, there are no baseline comparisons with nonlinear regression models. It is not clear how the ICA baseline, which could extract multiple linearly demixed source signals, is fairly compared. Is the ICA component shown in the figures and the best plots the best matched?  ICA is not only a linear approach but it also instantaneous. It doesn't consider past (or future values). Why couldn't MSE be used instead of cosine similarity? Is the model not able to match the mean or magnitude of the target?

1. Much of the time-series aspects are ignored. 
a) The paper's architecture does consider nor impose the causal structure on the decoder. That is the self attention creates a non-causal representation.
b) The effect of the length of the sequences in the regression performance is not discussed. Are all training sequences the same length? Can they vary in testing? 
c) Point about comparing against instantaneous ICA.

2. The EEG-EOG Dataset introduces an inaccurate description of electrophysiology. The electric potentials are instantaneously mixed. The reference cited for 'superimposed nonlinearly' actual refers to the non-linear dynamics not the combination of multiple sources impinging on the electrodes.  While nonlinearities can appear due to the sensing being nonlinear (amplifiers being saturated), this is not the sense of the description. 

3. In the ablation study, it appears the Seq2Seq method performs well without the variation inference. However some gap exists and it isn't clear if this could be lowered by keeping the affine transform from the encoding and still regularizing the difference from mean 0.

Minor points:
page 1, "proposal mostly focus" -> "proposal mostly focus"
page 2, "sour extraction"
Figure 2, where are the time indices in the representation fo the encoder and prior distribution? 
page 3, after equation 1 shouldn't the phrasing be that the KL divergence can be minimized by maximizing the variational lower bound?
page 5, the subscript notation of $1:T$ is not introduced and would seem to be a matrix. Generally the notation follows MATLAB/Octave but it is not clear to the reader.  For instance in equation 13, I am guessing the softmax operates on elements of the outer product by normalizing across each row.  The concatenation notation used in 14 should be defined. 
page 5, first line "To be realized
page 5, "forward bi-directional" ? 
page 5, $L^2$ distance is actually the squared $L_2$ distance between the signals (or $\ell_2$ in $\mathbb{R}^T$)
summary_of_the_review: While the paper presents a meaningful architecture for the task, key aspects of the nature of time-series are ignored and no meaningful baselines are used for benchmarking. Thus the paper is preliminary and needs a more thorough set of comparisons against other non-linear regression problems with and without causal restrictions on the modeling. 

-----------------------------

The authors have performed additional comparisons that help set the context. I am willing to raise my score if the text is further clarified to include the fact that is this is a non-linear multiple input and single output system identification model. 

I agree the concerns with non-causality may not be valid in all settings, and the additional testing on performance at shorter time segments is important.

Finally, I would like to encourage the revision to have careful wording regrading the non-linearity of the mixing versus non-linearity of processes. These two can get conflated. I agree that sometimes electrical recordings of linearly mixed sources result in nonlinear mixtures due to non-stationaries due to sensor/electrode movement as well as amplifier/recording nonlinearities. 

Further empricial analysis into the performance under challenging non-linearities such as clipping would be encouraged.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Authors propose a supervised variational component decoder (sVCD) framework that estimates a single source signal sequence from non-linear mixtures. Proposed model relies on a sequence-to-sequence translating variational encoder-decoder architecture, where optimization is performed based on a variational lower bound on the source signal’s data likelihood. Experiments are performed on artificially generated nonlinear sequence mixtures, RF sensing data and an EEG-EOG dataset.
main_review: The paper is well written and clear to understand. Proposed method is an elegant extension of a seq-to-seq VAE that exploits an attention mechanism, and reconstructs a source signal (that one needs to have available as a ground truth in the training set -which is a limitation-), based on a variational lower bound on the source signal’s data likelihood. Ablation studies are interesting and nicely performed, as they demonstrate a significant benefit of using an attention mechanism for time-series signal source extraction. In my opinion the paper’s main weakness is in the experimental analyses and evaluations that are presented.

Authors tried to make a distinction between nonlinear “source separation” and “source extraction” from mixtures. I am not fully convinced on the arguments around this theme. (1) For instance authors present arguments on “significantly reducing the computation cost”, when one is interested only in extracting the source signal of interest (i.e., nonlinear source extraction), rather than performing nonlinear blind source separation. This should be supported later in the experiments (which the authors did not), by discussing computation costs of the proposed algorithm, with respect to e.g., ICA. (2) Another line of argument by the authors state that “existing work on representation disentanglement focuses on image or video data, where signal mixtures never take place”. I do not fully agree on this statement, as in most cases these architectures are also applicable to time-series signal recordings (cf., studies on variational speech separation). To support their claim, authors can maybe present a baseline comparison to a simple VAE-based source separation model.

Performed experiments on the EEG-EOG dataset do not make much sense from a practical viewpoint, and reveal an important weakness. From what the reader can understand, the ground truth “y” to train the sVCD models for reconstruction is given as the EOG signals that are provided in the dataset. This means that sVCD recovers the best estimate of the EOG time-series from nonlinear EEG mixtures. While it is successfully shown that sVCD can perform this, it is not a meaningful experiment to demonstrate. In nonlinear EEG source separation/extraction, one is always interested in recovering some EEG source signal (e.g., originating from an independent cortical source component), that is clean from artifactual (e.g., EOG, EMG) mixing. I understand that since there is no given ground truth of any EEG source component, it would also be not possible to demonstrate such an experiment with sVCD. However, I do not understand why recovering EOG artifacts from EEG signals would be an interesting demonstration to include in the paper.

Overall, methodological comparisons seem shallow. It is also not clear how did the authors implement ICA (as a comparison baseline) in their experiments? How many independent components were initially extracted and how were the ICs selected following the decomposition? No supplementary or further descriptions of these experiments are presented. For instance, for EEG-EOG experiments, if one decomposes multi-channel EEG into independent components by pooling the training data, it should be generally obvious to recover the EOG component in one of the ICs besides cortical EEG signal source estimates (this is a widely-used approach for EOG artifact removal to discard ICs relevant to eye-blinks [T.-P. Jung et al., NIPS 1997], [S. Makeig et al., PNAS 1997]).

Minor comments:
- How did the authors determine the KL-term regularization strength? It is also not mentioned what are the parameters used?
- “sour” -> “source” on page 2, fourth line.
- “realized” -> “realize” on page 6, first sentence.

summary_of_the_review: The paper is clearly written and organized. However the paper is rather weak in depicting its contributions with the present experimental analyses and evaluations, and lacks detailed comparisons to other methods. I listed my major concerns in the main review, and would be willing to re-address my rating based on the authors’ responses and revisions.
review_title: null

Meta-review: This work has generated a lot of discussion between authors and reviewers and among reviewers.
Overall it is reported that the results on EEG are not conclusive and directly relevant for this field.
Besides the theoretical contribution is not reported as a strong point of the work and the
comparison with alternative baseline methods is judged too limited.

For all these reasons the paper cannot be endorsed for publication at ICLR this year.
Decision: Reject