Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
The idea of ICA is constructing a mapping from dependent inputs to outputs (=the derived features) such that the outputs are as independent as possible. As the input/output densities are often not known and/or are intractable, natural independence measures such as mutual information are hard to estimate. In practice, the independence is characterized by certain functions of higher order moments -- leading to several alternatives in a zoo of independence objectives.  

The current paper makes the iteresting observation that independent features can also be computed via adversarial objectives. The key idea of adversarial training is adapted in this context as comparing samples from the joint distribution and the product of the marginals. 

Two methods are proposed for drawing samples from the products of marginals. 
One method is generating samples but permuting randomly the sample indices for individual marginals - this resampling mechanism generates approximately independent samples from the product distribution. The second method is essentially samples each marginal separately. 

The approach is demonstrated in the solution of both linear and non-linear ICA problems.

Positive:
The paper is well written and easy to follow on a higher level. GAN's provide a fresh look at nonlinear ICA and the paper is certainly thought provoking. 


Negative:
Most of the space is devoted for reviewing related work and motivations, while the specifics of the method are described relatively short in section 4. There is no analysis and the paper is 
somewhat anecdotal. The simulation results section is limited in scope. The sampling from product distribution method is somewhat obvious.


Questions:

- The overcomplete audio source separation case is well known for audio and I could not understand why a convincing baseline can not be found. Is this due to nonlinear mixing?
As 26 channels and 6 channels are given, a simple regularization based method can be easily developed to provide a baseline performance, 


- The need for normalization in section 4 is surprising, as it obviously renders the outputs dependent. 

- Figure 1 may be misleading as h are not defined 

review_title: Thought provoking paper but lacks more detailed analysis

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a GAN variant for solving the nonlinear independent component analysis (ICA) problem. The method seems interesting, but the presentation has a severe lack of focus.

First, the authors should focus their discussion instead of trying to address a broad range of ICA problems from linear to post-nonlinear (PNL) to nonlinear. I would highly recommend the authors to study the review "Advances in Nonlinear Blind Source Separation" by Jutten and Karhunen (2003/2004) to understand the problems they are trying to solve.

Linear ICA is a solved problem and the authors do not seem to be able to add anything there, so I would recommend dropping that to save space for the more interesting material.

PNL ICA is solvable and there are a number of algorithms proposed for it, some cited already in the above review, but also more recent ones. From this perspective, the presented comparison seems quite inadequate.

Fully general nonlinear ICA is ill-posed, as shown already by Darmois (1953, doi:10.2307/1401511). Given this, the authors should indicate more clearly what is their method expected to do. There are an infinite number of nonlinear ICA solutions - which one is the proposed method going to return and why is that relevant? There are fewer relevant comparisons here, but at least Lappalainen and Honkela (2000) seem to target the same problem as the proposed method.

The use of 6 dimensional example in the experiments is a very good start, as higher dimensions are quite different and much more interesting than very commonly used 2D examples.

One idea for evaluation: comparison with ground truth makes sense for PNL, but not so much for general nonlinear because of unidentifiability. For general nonlinear ICA you could consider evaluating the quality of the estimated low-dimensional data manifold or evaluating the mutual information of separated sources on new test data.

Update after author feedback: thanks for the response and the revision. The revision seems more cosmetic and does not address the most significant issues so I do not see a need to change my evaluation.
review_title: Interesting nonlinear ICA method, but unfocused presentation and poor comparisons

final_score: rating: 3: Clear rejection
review_content: The focus of the paper is independent component analysis (ICA) and its nonlinear variants such as the post non-linear (PNL) ICA model. Motivated by the fact that estimating mutual information and similar dependency measures require density estimates and hard to optimize, the authors propose a Wasserstein GAN (generative adversarial network) based solution to tackle the problem, with illustrations on 6 (synthetic) and 3-dimemensional (audio) examples. The primary idea of the paper is to use the Wasserstein distance as an independence measure of the estimated source coordinates, and optimize it in a neural network (NN) framework.

Although finding novel GAN applications is an exciting topic, I am not really convinced that ICA with the proposed Wasserstein GAN based technique fulfills this goal.
 
Below I detail my reasons:

1)The ICA problem can be formulated as the minimization of pairwise mutual information [1] or one-dimensional entropy [2]. In other words, estimating the joint dependence of the source coordinates is not necessary; it is worthwhile to avoid it.

2)The PNL ICA task can be efficiently tackled by first 'removing' the nonlinearity followed by classical linear ICA; see for example [3].

3)Estimating information theoretic (IT) measures (mutual information, divergence) is a quite mature field with off-the-self techniques, see for example [4,5,6,8]. These methods do not estimate the underlying densities; it would be superfluous (and hard).

4)Optimizing non-differentiable IT measures can computationally quite efficiently carried out in the ICA context by e.g., Givens rotations [7]; differentiable ICA cost functions can be robustly handled by Stiefel manifold methods; see for example [8,9].

5)Section 3.1: This section is devoted to generating samples from the product of the marginals, even using separate generator networks. I do not see the necessity of these solutions; the subtask can be solved by independently shuffling all the coordinates of the sample.

6)Experiments (Section 6): 
i) It seems to me that the proposed NN-based technique has some quite serious divergence issues: 'After discarding diverged models, ...' or 'Unfortunately, the model selection procedure also didn't identify good settings for the Anica-g model...'.
ii) The proposed method gives pretty comparable results to the chosen baselines (fastICA, PNLMISEP) on the selected small-dimensional tasks. In fact, [7,8,9] are likely to provide more accurate (fastICA is a simple kurtosis based method, which is 
a somewhat crude 'estimate' of entropy) and faster estimates; see also 2).

References:
[1] Pierre Comon. Independent component analysis, a new concept? Signal Processing, 36:287-314, 1994.
[2] Aapo Hyvarinen and Erkki Oja. Independent Component Analysis: Algorithms and Applications. Neural Networks, 13(4-5):411-30, 2000. 
[3] Andreas Ziehe, Motoaki Kawanabe, Stefan Harmeling, and Klaus-Robert Muller. Blind separation of postnonlinear mixtures using linearizing transformations and temporal decorrelation. Journal of Machine Learning Research, 4:1319-1338, 2003.
[4] Barnabas Poczos, Liang Xiong, and Jeff Schneider. Nonparametric divergence: Estimation with applications to machine learning on distributions. In Conference on Uncertainty in Artificial Intelligence, pages 599-608, 2011.
[5] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Scholkopf, Alexander Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research, 13:723-773, 2012.
[6] Alan Wisler, Visar Berisha, Andreas Spanias, Alfred O. Hero. A data-driven basis for direct estimation of functionals of distributions. TR, 2017. (https://arxiv.org/abs/1702.06516) 
[7] Erik G. Learned-Miller, John W. Fisher III. ICA using spacings estimates of entropy. Journal of Machine Learning Research, 4:1271-1295, 2003.
[8] Francis R. Bach. Michael I. Jordan. Kernel Independent Component Analysis. Journal of Machine Learning Research 3: 1-48, 2002.
[9] Hao Shen, Stefanie Jegelka and Arthur Gretton. Fast Kernel-Based Independent Component Analysis, IEEE Transactions on Signal Processing, 57:3498-3511, 2009.

review_title: Proposed Wasserstein GAN: not well-suited to ICA

Meta-review: The paper proposes the use of GANs to match the joint distribution of features to the product of their marginals for ICA. The approach is totally plausible but reviewers have complaints about lack of rigor and analysis in terms of (i) mixing conditions under which the proposed GAN based approach will work, given that ICA is ill-posed for general nonlinear mixing  (ii) comparison with prior work on linear and PNL ICA.

Further, in most scenarios where GANs are used, one of the distributions is fixed (say, the real distribution) and the other is dynamic (fake distribution) trying to come close to the fixed distribution during optimization. In the proposed method, the discriminator encodes the distance b/w joint and product of marginals which are both dynamic during the learning. It might be useful to comment whether or not it has any implications wrt increased instability of training, etc. 
Decision: Reject