Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

Summary:
 

This paper formulates blind source separation as a statistical estimation problem in a log-linear model and solves the statistical problem by a modified natural gradient method. 


##########################################################################

Reasons for score: 
 

This is a complete work. However, this paper simply presents the method and experimental results and an in-depth discussion on why the proposed method should be superior than existing ones is lacking, so I do not feel this work meets the standard of ICLR. There are some misconceptions in the arguments of this paper (see cons). 


##########################################################################Pros: 
 

Pros: 

1. The idea of formulating blind source separation as statistical estimation in a log-linear model seems to be novel. 
 

##########################################################################

Cons: 
 

1. The novelty seems limited. The main novelty is in formulating the blind source separation problem as a statistical estimation problem. The log-linear model and natural gradient method seems to be already known to experts.  

2. The presentation is highly imbalanced. While the log-linear model and natural gradient method are already known, this paper spent most of the space on stating the known results and lacks an in-depth discussion on why the proposed algorithm should be superior.   The main argument supporting the proposed algorithm is that non-convex optimization is bad and convex optimization is good. This is apparently not true in general. 

3. The paper argues that existing non-convex approaches are not favorable because they are sensitive to initializations. This is in general not true. The authors should show empirically that sensitivity w.r.t. initializations actually poses an issue in existing approaches, or give appropriate citations supporting their claim.  

4. The paper also argues that existing approaches are not favorable because they impose strong assumptions on the problem. However, I think the log-linear model assumption in this paper is not obviously weaker than the assumptions in existing approaches. 

5. This paper proposes using natural gradient descent to solve the convex optimization problem, because the iteration complexity of natural gradient descent is low. However, a low iteration complexity is not very meaningful; what’s meaningful is for example the overall computation complexity to achieve an \varepsilon-approximate solution.  

6. To reduce the per-iteration complexity of natural gradient descent, this paper proposes a seemingly heuristic variant in (6) and (7) and claims the algorithm also converges to the optimum. I do not see why the claim is true.   

7. The presentation can be improved. The introduction to the log-linear model and natural gradient descent is quite long but not very readable to a non-expert in the two topics. It is better to have this paper checked by a native speaker, though the number of typos and grammar mistakes is acceptable.  

8. The title features “Legendre transformation,” but I do not find it in the main text.  

9. Typo: The set \mathfrak{B} as defined in Section 2.3 is not exactly the set of all probability distributions. 
 

##########################################################################

After reading the rebuttal: 
 

I keep my score. The authors should rethink how to claim the novelty in a concrete way. The presentation needs to be improved for readers not familiar with information geometry (they should be the majority). I do not get 6) in the rebuttal. 
 

#########################################################################
review_title: Weak discussion on the superiority of the proposed method

final_score: rating: 4: Ok but not good enough - rejection
review_content: Additional comments: I have read the authors' response and the other reviews. While my initial concerns have been mostly addressed, there are still concerns from the other reviews and I have revised my score accordingly.

In addition, I am not completely satisfied with the authors' response concerning higher-order interactions. In the end, even if you add other nodes, the operations are simply matrix operations and so you can model any higher order interactions with a single matrix. This should be clarified further.

______________________________________________________________________________
This paper uses an information geometry-based model to perform blind source separation. The problem is definitely significant and should be of interest to conference attendees. The approach appears to overcome many of the weaknesses in existing approaches and outperforms them in selected experiments. 


Pros:
IGBSS, the proposed approach, has many advantages such as convexity and potential nonlinearity allowing it to adapt to nonlinear mixing effects. The empirical results shown here also look good.

The paper appears to be technically correct.

Cons:
The experiments seem pretty simple. Only a few experiments are performed. While the results are fairly convincing, it seems like a lot more experiments could have been performed with summary statistics reported. 

At the bottom of page 3, it's stated that the partial ordering allows for higher order interactions. This isn't clear to me as it still appears that only matrix operations are being performed with this model and thus only first order interactions can be modeled. Can the authors clarify this?

I would be willing to raise my score if these issues were properly addressed.

Other comments:
In Table 1, FastICA performs best wrt RMSE in experiment 2 with third order interactions and wrt SNR in experiment 2 with second order interactions. The authors should fix this.

FastICA tends to do better wrt RMSE and SNR as the order increases. Do the authors have any insight into why this is?
review_title: An interesting use of information geometry in blind source separation

final_score: rating: 2: Strong rejection
review_content: We find he paper inappropriate for ICLR for many reasons. We give below some elements to help the authors in increasing the relevance of their work and paper.

The title is inappropriate, as Legendre transformation is not used in this work. And the word Legendre is never used in the paper. 

The introduction is too long, describing many elements that are not necessary for the paper. For example, the paragraph describing the PCA and some of its variants, as well as the orthogonality constraint. In the paper, the PCA is not considered, neither the orthogonality constraint.

The authors focus on the fact that other methods are sensitive to initialization and they may converge to local optimum. In practice, the proposed method may have similar issues, as can be seen from algorithm 1 because it requires the initialization of \theta_s. It is not clear what is the used initial values, but it seems that it is 0 and not random because it makes the obtained results constant after 49 runs. This doesn’t mean that it is independent of the initialization. 

It is easy to demonstrate that the proposed method cannot jointly verify the independence on initialization and the convexity/convergence to global optimum. To show this, we can initialize with the results obtained from FastICA. As FastICA sometimes outperforms all the other methods, this means that the proposed method should provide better results that FastICA throughout iterations, because the optimization problem is supposed to be convex and it converges to the global optimum; However, this means that the method depends on the initialization. If the proposed method is supposed to be invariant to initialization, this means that it will yield the same values given in the paper, thus not outperforming FastICA is several cases. This means that it converges to an optimum that is not a globale one, as FastICA converges to better optimum solution.

Experiments are not convincing, because comparing only to too old methods, such as FastICA from 2000 and dictionary learning from 1997. The authors need to compare to more recent methods from the state of the art.

There are many spelling and grammatical errors, such as disctionary.
review_title: Paper not acceptable

final_score: rating: 3: Clear rejection
review_content: %% post-rebuttal %%
Though I appreciate the efforts of the authors to clarify their methodology and assumptions in their answer, these clarifications (which I still don't fully grasp or agree with) have not been reflected in the revised version. This work still needs a significant revision and, in my opinion, cannot be accepted in its current form.
%%%%%%%%%%%%


0) I reviewed an earlier version of this paper. The ICLR submission is similar to the earlier one, hence the similar (but updated) review. Generally, the paper is hard to follow and can only be understood by someone well-versed in both source separation and information geometry.

1) Presentation of the ideas is lacking (insufficient explanations, motivations) and I failed to understand the main contribution of the paper, i.e., how the concept of “information geometric log-linear model” can be applied to BSS. The log-linear model models a discrete probability distribution of ordered events. As such the authors introduce a specific ordering for BSS that I failed to understand. Eq. (2) plays a central role but details and motivation are lacking and it’s hard to understand why this ordering is interesting and why a log-linear model makes sense.

- Exemples would be welcome in Section 2.1. What is Omega ? what is an s ? what is S ? what is the meaning of Psi ? what is the intuition behind (1) ? 

2) I would encourage the authors to clarify the settings in which their methods is applicable. It seems to be able to tackle undetermined methods and some forms of nonlinearity, though it's not clear why. 
- What are the assumptions about the sources and mixtures ?
- How come your method can identify the sign of the sources ? What sort of prior information is available ?
- The preprocessing of real-valued data (basically applying exp or some form of normalisation) is disturbing.

3) The experiments consider artificial mixture of images and artificial mixtures of synthetical signals with little practical significance.
- Can you explain more clearly why ICA or NMF fail on the simple 3x3 image separation problem ? 
- Do you have any idea of an application that could benefit from the nonlinear model in Section 3.2 ?

4) The introduction of the paper could be improved as it seems to reveal some misconceptions about blind source separation (BSS). For example PCA is not a method dedicated to the separation of orthogonal sources (such a problem is actually not identifiable). It is instead often used as a pre-processing in ICA (whitening). The introduction also sustain a confusion between feature extraction and source separation. These two different tasks can be handled by similar methods (ICA, NMF) but in very different settings. In source separation, the data X contains signal in rows (e.g., the cocktail party problem). In feature extraction, the focus is on the columns of X which contains data samples (such as short-time frames of a single-channel mixture). 

review_title: Hard to follow, important information missing

Meta-review: The focus of the submission is blind source separation (BSS). The authors propose a log-linear model based formulation to tackle the task and to relax assumption/restrictions (linear mixing, non-convex objective, ...) present in previous techniques. They use the maximum likelihood approach [Eq. (3)] with natural gradient descent for optimization, and illustrate the efficiency of the approach in two toy examples (separation of mixed images and that of sin/sign/sawtooth signals).

BSS is an important task in machine learning with various applications. As assessed by the reviewers, however, the submission is in a quite preliminary stage:
(i) Section 1 is rather long, still it lacks providing relevant context to the work.
(ii) The introduction of the main ideas/motivation, the assumptions imposed, and the explanation of the notations are missing.
(iii) The usefulness of the proposed approach is questionable; the demos focus on artificial toy examples.
More work and significant revision are needed before publication.
Decision: Reject