Hierarchical Probabilistic Model for Blind Source Separation via Legendre Transformation
Authors: Simon Luo
Lamiae Azizi
Mahito Sugiyama
Venue:  (2021)
Keywords: blind source separation
log-linear model
energy-based model
information geometry

Abstract: We present a novel blind source separation (BSS) method, called information geometric blind source separation (IGBSS). Our formulation is based on the log-linear model equipped with a hierarchically structured sample space, which has theoretical guarantees to uniquely recover a set of source signals by minimizing the KL divergence from a set of mixed signals. Source signals, received signals, and mixing matrices are realized as different layers in our hierarchical sample space. Our empirical results have demonstrated on images and time series data that our approach is superior to well established techniques and is able to separate signals with complex interactions.

Under review as a conference paper at ICLR 2021
HIERARCHICAL PROBABILISTIC MODEL FOR BLIND SOURCE SEPARATION VIA LEGENDRE TRANSFORMATION
Anonymous authors Paper under double-blind review
ABSTRACT
We present novel blind source separation (BSS) method, called information geometric blind source separation (IGBSS). Our formulation is based on the log- linear model equipped with a hierarchically structured sample space, which has theoretical guarantees to uniquely recover set of source signals by minimizing the KL divergence from a of mixed signals. Source signals, received signals, and mixing matrices are realized as different layers in our hierarchical sample space. Our empirical results have demonstrated on images and time series data that our approach is superior to well established techniques and is able to separate signals with complex interactions.
INTRODUCTION
The objective of blind source separation (BSS) is to identify set of source signals from set of multivariate mixed signals" BSS is widely used for applications which are considered to be the
"cocktail party problem". Examples include image/signal processing (Isomura & Toyoizumi 2016).
artifact removal in medical imaging (Vigário et al.) 1998). and electroenceehalogran (EEG) signal separation (Congedo al. 2008). Currently, there are number of solutions for the BSS problem.
The most widely used approaches are variations of principal component analysis (PCA) (Pearson 1901 Murphy 2012) and independent component analysis (ICA) (Comon 1994 Murphy 2012).
However, they all have limitations with their approaches.
PCA and its modern variations such as sparse PCA (SPCA) Zou al., 2006), non-linear PCA (NLPCA) (Scholz et al., (2005), and Robust PCA (Xu et al. 2010) extract specified number of components with the largest variance under an orthogonal constraint, which are composed of a linear combination of variables. They create set of uncorrelated orthogonal basis vectors that represent the source signal. The basis vectors with the largest variance are called the principal components and is the output of the model. PCA has shown to be effective for many applications such as dimensionality reduction and feature extraction. However, for BSS, PCA makes the assumption that the source signals orthogonal, which is often not the in most practical applications.
Similarly, ICA also attempts to find the N components with the largest variance, but relaxes the orthogonality constraint. All variations of ICA such infomax (Bell & Sejnowski] 1995), Fas- 1CA (Hyvärinen & Oja 2000) and JADE (Cardoso 1999) separate multivariate signal into addi- tive subcomponents by maximizing statistical independence of each component. ICA assumes that each component is non-gaue san-gaussian and the relationship between the source signal and the mixed signal an affine transformation. addition these assumptions, ICA is sensitive to the initialization of the weights as the optimization is non-convex and is likely to converge to a local optimum.
Other potential methods which can perform BSS include neg-nege matrix factorization (NMF) (Lee & Seung, [001]; [erne et al., (007), dictionary learning (DL) (Olshausen & Field 1997.. and reconstruction ICA (RICA) (Le et al. 201T). NMF, DI. and RICA are degenerate ap- proaches to recover the source signal from the mixed signal. These approaches are more typically used for feature extraction. NMF factorizes a matrix into two matrices with nonnegative elements representing weights and features. The features extracted by NMF can be used to recover the source
Mixed signals and received signals used exchangeahly throughout this article.

---

Under review as a conference paper at ICLR 2021
signal. More recently there more advanced techniques that uses Short-time Fourier transform (STFT) to transform the signal into the frequency domain to construct a spectrogram before apply- ing NMF (Sawada et al. 2019). However, NMF does not maximize statistical independence which is required to completely separate the mixed signal into the source signal, and it is also sensitive to initialization the optimization is non-convex. Due to the non-convexity, additional constraints or heuristics for weight initialization is often applied to NMF to achieve better results (Ding et al.) 2008 Boutsidis & Gallopoulos] 2008). DL can be thought of as a variation of the ICA approaches which requires an over-complete basis vector for the mixing matrix. DL may be advantageous be- cause additional constraints such as positive code or dictionary can be applied to the model. However, since it requires an over-complete basis vector, information may be lost when reconstruct- ing the source signal. In addition, like all the other approaches, DL is also non-convex and it is sensitive to the initialization of the weights.
All previous approaches have limitations such as loss of information or non-convex optimization and require constraints or assumptions such as orthogonality or an affine transformation which are not ideal for BSS. In the following, we introduce our approach to BSS, called IGBSS (Information Geo- metric BSS), using the log- linear model (Agresti| 2012), which can introduce relationships between possible states into its sample space (Sugiyama et al.J 2017). Unlike the previous approaches, our proposed approach does not have the assumptions or limitations that they require. We provide flex- ible solution by introducing a hierarchical structure between signals into our model, which allows us to treat interactions between signals that are more complex than an affine transformation. Unlike other existing methods, our approach does not require the inversion of the mixing matrix and is able to recover the sign of the signal. Thanks the well-developed information geometric analysis of the log-linear model (Amari) 2001). optimization of our method is achieved via convex optimization, hence it always arrives at the globally optimal unique solution. Moreover, we theoretically show that it always minimizes the Kullback-Leibler (KL) divergence from a set of mixed signals to set of source signals. Our experimental results demonstrate that our hierarchical model leads to better separation of signals including complex interaction such as higher-order feature interactions (Luo & Sugiyama 2019) than existing methods.
2 FORMULATION
BSS is formulated as a function f that separates set of received signals X into a set of source signals Z, i.e., 7= = f(X). For example, if one employs ICA based formulation, the BSS problem reduces to X = AZ, where the received signal X E R 4x with L signals with the sample size M is affine transformation of the source signal Z E R N with N signals and a mixing matrix A E L The objective is to estimate Z by learning A given X. Our idea is to use the log-lineat model (Agresti| 2012) which is a well- energy-based model, to take -ff-nn transformation into account and formulate BSS as a convex optimization problem.
2.1 LOG-LINEAR MODEL ON PARTIALLY ORDERED SET
We use the log-linear model given in the form of
log p(w) = E1ssw0--
(I)
SES
where p(w) E (0, 1) is probability of each state w E Ω and S C S is a parameter space such that parameter value 8g R is associated with each S, and vi() is the partition function that Ewen p(w) = 1. In this formulation, we assume that the set Q of possible states, equivalent to the sample space in the statistical sense, is a partially ordered set (poset); that is, it is equipped with a partial order "=" (Gierz et al. (003) and 1س58 = if 8 A w and O otherwise. This for- al.) (016) and used to model the matrix balancing problem (Sugiyama et al.) 2017). which includes Boltzmann machines as a special case (Luo & Sugiyama 2019). If we index 52 as Ω {W), W2,..... ... w]p)) we obtain the following matrix form:
mulation is firstly introduced by /Sugiyama
log F0- (0),
where p E with pt p(wx), 0 R [2) such that 0. Ow; if wi E S and 0g otherwise, F (fz) E (0, 1) | with fij Lيy and cb( (1b(0), -- (b)) E R Each vector
2

---

Under review as a conference paper at ICLR 2021
is treated column vector, and log is entry-wise operation. This matrix form is often used general form of the log-linear model (Coull & Agresti 2003) and F is called a model matrix, which represents relationship between states. The assumption to the log-lineas model is that F is needed to be non-singular, and Sugiyaaa et al. (2017) showed that Equation (T) with a poset ft always provides non-singular model matrix; that is, F is regular long each entry is given as fij Jwys This property is powerful in mathematical modeling as we can introduce any partial order structure into S2, which we will use to introduce our hierarchical structure in tne next subsection solve BSS.
2.2 LAYER CONFIGURAIIN FOR BLIND SOURCE SEPARATION
Our key idea is to introduce a hierarchical layered structure into the sample space Ω of the log-linear model achieve BSS. We call this model information geometric BSS (IGBSS) as its optimality is supported by the tight connection between the lig-linear model and information geometric property of the space of distributions (statistical manifold), which will be shown in the next subsection. We implement three layers of BSS, the mixing layer, the source layer, and the received layer, into Ω as partial orders and learn the joint representation on it using the log-linear model. The received layer and the source layer represent the input received signal and the output source signal of BSS, respectively, and the mixing layer encodes information of how to mix the source signal. In the following, consistently assume that L is the number of received signals, M is the sample size, and N is the number of source signals.
Let us construct three layers in the sample space Q as = [1]
an a2l
d12
211
#12
211
212
U 2 X with A = {an1..... CIN ) 2 (=11--.., 2NM), and X' [I FLM The element 1 denotes the least element, and it acts as partition function and 0, (0) always holds. We use 2D indexing of elements in each layer to make the correspondence between our formula- tion and ICA based formulation clear; that is, these three layers A, 2, and X' are analogue to a mixing matrix A E R a source matrix Z E TR N and a received matrix X E BLXM respectively We will also use symbols w and to denote elements of Ω i.e. they can be 1. Atn. Zum, and Zim- is always assumed that the parameter space of the log- linear model S = AU 2 C Ω, meaning that mixing and source layers are used as parameters repre- sent distributions in our model. Here we introduce a partial order = between layers. Define
022
221
222
=
#21
222
Mixing layer
Source layer
Received layer
J111
0:2
3/22
Figure 1: An example of our sample space. Dashed lines show removed partial orders to allow for learning.
aij I Zzy'' aij B Z3'"
ifj=', otherwise,
Zij Byy Zij A Tays
if otherwise
(2)
for each element in three layers A, Z, and X, and we do not any ordering among elements in the same layer. Since it is a partial order, transitivity always holds, for example, d11 I 322 as 411 I 412 and 212 I T22- The first condition encodes the structure such that the source layer is higher than the mixing layer, and the second condition encodes that the received layer is higher than the source layer. An example of our sample space with L M = N 2 is illustrated in Figure I
The joint distribution for BSS is described by the log-linear model in Equation 1 over the sample space Ω = {1} U A U Z U X equipped with the partial order defined in Equation (2). If we learn the joint distribution from received signal X, we will obtain probabilities on the source layer p(Z11)... p( which represents normalized source signals. The rational of our approach is given as follows: The connections between each layer is structured so that the log- log-linear model performs a similar computation with the ICA based approach X AZ. Our structure ensures that each p(zmm) is determined by (OE))))E(N)) and (0 (0=mpn) )NE[N] with [N] = {l,..., always have Oin. < Tim and Zym < Tim- Moreover, this formulation allows us to model more complex interaction than affine transformation, such as higher-order interactions, between signals if
N.....,N. NN. as we
We abuse an entry Zim of X and its corresponding state in X' avoid complicated notations.
3

---

Under review as a conference paper at ICLR 2021
additionally include partial order structure into 2 and/or A, which cannot be treated by simple matrix multiplication.
2.3 OPTIMIZATION
We train the log-linear model by minimizing the KL divergence from an empirical distribution p. which is identical to the normalized received signal X E IRL to the model joint distribution given by Equation [ or, equivalently, maximizing the likelihood. More precisely, we normalize a given X by dividing each entry by the sum of all entries; that is, an empirical distribution p
obtained as p(Zim) = Ftm/ Lt.m Jim- If X contains negative values, an exponential kernel exp (4 exp (Iim.) or min-max normalization (emmeemin(XX) (XX)))((mxxXXXX)) (X) (eixx>emmmx))) min(X)) can used, where e is some arbitrary small value to avoid zero probability. We also assume that p(amn) = 0 and p(znmm = 0 for all O/n E A and Zmom 2. The objective function is given as
f(w)
arg min DKL (øllp) = arg min PEPo
Σ ((w) log
p(w)'
PEB.
WES
where Fo is the set of distributions that can be represented by Equation sample space Ω = (UJ 2 U X and S Au Z.
I with our structured
The remarkable property of our model is that this optimization problem convex and it is guaranteed that gradient-based methods can always arrive at the globally optimal unique solution. To show this, we analyze the geometric structure of the statistical manifold, the set of probability distributions, generated by the log-linear model. Let s 2 {1). First we introduce another parameterization (ww)e of the log- log-linear model, which is defined as
Tlw Σ 1wwsp(()) SEN
Note that 1) = 1 always holds and we do not include it into parameters. In addition, for theoretical consistency we change the parameter space used in Equation (I) from S to 22+ and assume that 0, = 0 if w & S. Again we do not include 0_ as a parameter as it is the partition function. Two parameters (fw)wEst and (we) have clear statistical interpretation as is widely known that any log-linear model belongs to the exponential family, where 0 and correspond to natural and expectation parameters, respectively. 0 and 1) are connected via a Legendre transformation which means that they are both differentiable and have a one-to-one correspondence. To simplify the notation, denote by 0 and the corresponding 0 and 17 of the empirical distribution p. Let . = (p O < p(w) < 1 for all w E []} be the set of all probability distributions. This set forms statistical manifold with dually flat structure, which is the canonical geometric structure in information geometry (Amaril 2016), with its dual coordinate system ((O))eeat (nw)wen-) ); that is. both of (0w) ( and (Nw JeE work as coordinate systems and determine distribution in Z. The Riemannian metric with respect to 0 is given as
Oq, Iss' 00
i log p(w) θ Ologp(())
E
(1 - NA7]''
(5)
00,
00.
WEB
which coincides with the Fisher information for natural gradient.
(Sugiyaaa al. 2017
Theorem 3) and we will use it
Now we consider two submanifolds Pa, P. B. which we define as
TB { p E 23 00,, = 0, Ve E &}, F = { p E Q n = n Vw E MM
E= S =
Note that this Ta coincides with that in Equation 3 The submanifold Ba is called an e-flat sub- manifold and F. an m-flat submanifold in information geometry. The highlight of considering these two types of submanifolds is that, if & M 1 and & U M Ωt it is theoretically guaranteed that the intersection Po n B. is always a singleton and is the optimizer of Equation (Amari 2009 Theorem 3), that is, it is the globally optimal solution of our model.
Optimization is achieved by e-projection, which seeks Bo 0 B. in the e-flat submanifold Ty. The e-projection is always convex optimization Bo is convex with respect 0 this is because is
4

---

Under review as a conference paper at ICLR 2021
Algorithm Information Geometric BSS
1: Function IGBSS(X, S):
2: Compute p from
3: Compute 1) (fss) from p
4: Initialize (0.).ES (randomly or 0g
5: repeat
6: Compute using the current parameter (0,) (0,)sES
7:
Compute (Ms)sES from p
8:
(ATw)wEZ +- (Nw)EE - (Ww)
9:
(ATW)WEE f (WE - (EE)
10:
Compute the Fisher information matrix for source layer Gz and the mixing layer G,
(0w)EEZ + (Bw)EEZ Z (Am))wEZ
(O=) + (Ow)
(WE
13:
until convergence of (0s)seS
14:
End Function
coordinate system of Po that is linearly constrained 0. We can therefore use the standard gradient descent strategy to optimize the log-linear model. The derivative of the KL divergence with respect 0, is known be the difference between expectation parameters 7) (Sugiyama et al. 2017 Theorem 2): (0/00 100, . (00,)Da (i.((i) () = !/s ns. and the KI. divergence DRL. (pl p) is minimized if and only if 1/2- fis for all S.
From our definition of in Equation (). we have 75. = T7 for all žkd, 21r E 2. Therefore all elements in the source layer will learn the same value. This problem be avoided by removing some of partial orders between source and received layers. We propose to systematically remove the partial order Zzj 3 zz if i' to ensure Yeu Tagr (see Figure while other strategies possible as long as Maa * Magi is satisfied, for example, random deletion of such orders.
Using the above results, gradient descent can be directly applied achieve Equation B.) However, this may need large number of iterations to reach convergence. To reduce the number of iterations, we propose use natural gradient (Amari, 1998), which is a second-order optimization approach and will also always find the global optimum. Let us re-index S A U 2 as S = {s1, $2, ... $15) IT.
and assume that 0 1 6, 1T and 7 ! ! In each step of natural gradient, the current f is updated to next by the following formula:
n 1
G - 1))
where G -gii) E [S Equation (5)
is the Fisher information matrix such that each Pij is given Iscsy in
Although the natural gradient requires much less iterations compared to the gradient descent, matrix inversion G-1 computationally expensive as has the complexity of O((S)). In addition, FIM values are often too small and optimization becomes numerically unstable. To solve these problems, we separate the update steps in the source layer and the mixing layer:
(0w.next)wEZ (Ow)EEz - ( (Anw)wEZ, (Ow, WEEA (fW)EEA G
(6)
where Gz and G are the Fisher information matrices for source and mixing layers, respectively. Note that this also leads to the same global optimum. They are constructed by assuming all the other parameters are fixed. This approach reduces the time complexity to +[A3) The full algorithm using natural gradient is given in Algorithm [ Computation of p from 0 and from can be achieved using Equations ( and (4) We also give more explicit description of and n for each layer in Appendix. The time complexity The complexity to compute An in Algorithm Line g and Line 9 is ( ) = ((S)).
compute p in Algorithm Line 6 is S(
Therefore the total complexity of each iteration O 3 LAAA% 3
5

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(c)
(d)
(e)
( NMF
(a)
(b)
(c)
(d)
(e)
(n) NMF
GT
Mixed
IGBSS
ICA
DL
GT
Mixed
IGBSS
ICA
DL
Figure 2: First-order interaction experiment.
Figure 3: Third-order interaction experiment.
Table I: Signal-to-Noise Ratio of reconstructed signal. (+) Results for Figure 2
(t) Results for
Figure Scores are means + standard deviation after 40 runs We have applied different weight initialization after each run.
Mean Squared (RMSB)
Signal to noise FasICA
Exp.
Order
IGBSS
FaslCA
DL
NMF
JGBSS
DL
NMF
First"
0.252
0.090
0.300 1 0.089
0,394
0.011
0.622
0.000
12,588
0.000
11,688
1 4,8299
5,800
0.008
1,704 1 0.000
1
Sccond
0.260
0.000
0.285
+ 0.096
0.421
0.000
0.602
= 0.000
10.729
0.000
12:353 + 4.255
(1.26
= 0.448
-3.426 : 0.000
Thind"
0.252
0.000
0,260
0.111
0.362
0.030
0,612 + 0.080
12,588
0.000
12.922
5,590
1,471 0.358
0.039
0.000
Hist
0.133
0.000
0.284
0.064
0.474
0.067
0.000
14.215
0.000
11.218
2058
2,140
Sepond
0.000 0.256
0.263 +
0.576
0.008
0.684
0.000
10,612
0.000
11:986
2.157 11,475
-1,589 -0.669
-3.675
Third
0.007
0.000
0.478 0.121
0.625
0.000
10.785
2.176
3
Second
0,200
0.000
0,280
0.049
0.515 0.007
0.709
0.000
10,862
0.000
10.171
1 2,353
0.529
0.228
-5.579 0.000
Third
0.203 0.000
0,239 :: 0.056
0,536 0.006
0.682 = 0.000
11,075
0.000
11.041 +: 2.708
-0.244 = 0.185
-4.61 #: 0,000
EXPERIMENTS
We empirically examine the effectiveness of IGBSS to perform BSS using real-world image and synthetic time-series datasets for affine transformation and higher-order interactions between signals, All experiments were run on CentOS Linux 7 with Intel Xeon CPU E5-2623 v4 and Nvidia QuadroGP100
3.1 BLIND SOURCE SEPARATION FOR AFFINE TRANSFORMATIONS ON IMAGES
In our experiments, we use three benchmark images widely used in computer vision from the Uni- versity of Southern California's Signal and Image Processing Institute (USC-SIPI which include "airplane (F-16)", "lake" and "peppers". Each image standardized to have 32x32 pixels with red, green and blue color channels with integer values between 0 and 255 to represent the intensity of each pixel. These images shown in Figure [a are the source signal which are unknown to the model. They are only used ground truth to evaluate the model's output. The equation X = AZ is used to generate the received signal X by randomly generating values for a mixing matrix A using the uniform distribution which generates real numbers between and 6. The images are then rescaled to integer values within the range between 0 and 255. The received signal X, which is the input to the model, is the three images shown in Figure 25 The three images for the mixed signal may look visually similar, however, they are actually superposition of the source signal with differ- ent intensity. The objective of our model is reconstruct the source signal Z without knowing the mixing matrix A.
We compare our approach to FastICA (Hyvärinen & Oja 2000) with the log cosh function as the signal prior, dictionary learning (DL) (Olshausen & Field 1997) with constraint for positive dic- tionary and positive code, and NMF with the coordinate descent solver and non-negative double singular value decomposition (NNDSVD) initialization (Boutsidis & Gallopoulos 2008) with zero values replaced with the mean the input.
'The code available in the supplementary material and will be publicly available online after the peer review process http://sipi.usc.eddddattt

---

Under review as a conference paper at ICLR 2021
Bacdled FHER Oegett
Figure 4: Time series signal experiment.
Since BSS is an unsupervised learning problem, the order of the signal is not recovered. We iden- tify the corresponding signal by taking all permutations of the output and calculate the minimum cuclidean distance with the ground truth. The permutation which returns the minimum error is con- sidered as the correct order of the image. The scale of the output is also not recovered, thereby have used min-max normalization to the output each model.
Separation results for images are shown in Figure 2 Our proposed approach IGBSS is able to recover majority of the "shape" of the source signal, while the intensity of each image appears to larger than the ground truth for all images. Small residuals of each image can be seen the other images. For instance, in the airplane (F-16) image, there residuals from the lake image can be clearly seen. Compared to the reconstruction of IGBSS with FastCCA, DL and NMF, IGBSS performs significantly better as all the other approaches are unable to clearly separate the mixed signal. FastCCA was unable provide reasonable reconstruction with mixed signal. To overcome this limitation of FastICA, we randomly generated another column of the mixing matrix and append to the current mixing matrix to create 4 mixed signals as an input to FastCCA to recover a more reasonable signal.
The root mean square error (RMSE) of the Euclidean distance and the signal-to-nots ratio (SNR) between the reconstruction and the ground truth is calculated quantify results of each method. The SNR is computed by SN = 20 10gio(2norm (top row for each experiment). In the table, we present three experiments with different RGB images from USC-SIPI dataset, for each experiment we generate a new mixing matrix, where the second and the third experiments uses images of "mandrill". "splash", "jelly beans" and "mandrill", "lake", "peppers", respectivel.. Ground truth and resulting images for second and third experiments presented in Supplement. Our results clearly show that IGBSS is superior other methods, that is, IGBSS has consistently produced the lowest RMSE error for every experiment. When looking at the SNR ratio, our model has produce the highest SNR for majority of the cases and is always able to recover the same result after each run as it is formulated as convex optimization.
(I znorm) )). The full results are shown in
3.2 BLIND SOURCE SEPARATION WITH HIGHER- FEATURE INTERACTIONS
We demonstrate the ability BSS for our model to include higher-order feature interactions in BSS. We use the same benchmark images in the standard BSS the source signal Z for experiment. We generate the higher-order feature interactions of the received signal by using the multiplicative product of the source signal. If we take into account up kth order interaction (k & N).
Σ
Σ Σ
ΣΣ Σ
Tim
Ain Zym
alning &ngm
Glningnn Zri m Zngm Zngm
Σ
Σ
Oiny En ... 2ngm-
All the other known approaches take into account only first order interactions (that is, affine transfor- mation) between features. Differently, our model can directly incorporate the higher-order features we do not have any assumption of the affine transformation. When consider up to keth der interactions, we additionally include elements corresponding to new mixing parameters into the mixing layer. For example, if k 2, nodes for ainjng are added and dining I Znm if ny or
7

---

Under review as a conference paper at ICLR 2021
Table Quantitative results for time-series separation experiment (mean = standard deviation with 40 runs).
(a) Root Mcan Squared Error (RMSE)
(h) Signal-to-noise (SNR) (units in dB)
Order
IGBSS (min--aa max)
IGBSS (exp)
FastCA
Order
IGBSS (min max)
IGBSS (exp)
FasiCAA
First
0.702 + 0.000
0.703 + 0.000
0.414 0.286
First
3.596 +: 0.000
3,600 + 0.000
15.391 = 3.813
Second
0 = 0.000
0.921 + 0.000
1,700 0.167
Second
0.291 0.000
0,042 0.000
-5.003 + 1.124
Third
0.967 + 0.000
0.961 + 0.000
1,388 0,178
Third
0.340 0.000
0,128 0.000
-3.427 1 1.249
f2 n. Figure shows experimental results for the third-order feature experiment. Our approach IGBSS shows superior reconstruction of the source signal to other approaches. All the other ap- proaches except for NMF is able achieve reasonable reconstruction. NMF is able to recover the "shape" of the image. however, unlike IBSS, NMF is a degenerate approach, so it is unable to re- cover all color channels in the correct proportion, creating discoloring for the image which is clearly shown in the SNR values. Since the proportion of the intensity of the pixel is not recovered. In terms of both of the RMSE and the SNR shown in Table [ IGBSS again shows the best results for both second- and third-order interactions of signals across three experiments.
3.3 TIME SERIES DATA ANALYSIS
We demonstrate the effectiveness of our model on time series data. In our experiments, we cre- ate three signals with 500 observations each using the sinusoidal function, sign function, and the sawtooth function. The synthetic data simulates typical signals from a wide range of applications including audio, medical and sensors. We randomly generate mixing matrix by drawing from uniform distribution with values between 0.5 and 2. In our experiment, we provide comparison of using both min-max normalization and exponential kernel as a pre-processing step and compare our approach with FastCCA.
Experimental results are illustrated in Figure These results show that IGBSS is superior to all the ICA approaches because it is able to recover both the shape of the signal and the sign of the signal, while all the other ICA approaches are only able to recover the shape of the signal and are unable to recover the sign of the signal. This means that ICA could recover flipped signal. We have paired the recovered signal of ICA with the ground truth by finding the signal and sign with the lowest RMSE error. In any practical application, this is not possible for ICA because the latent signal is unknown. Through visual inspection, IGBSS is able to recover all visual signals with high accuracy, while FastICA is only able recover the first-order interaction and it is unable to produce a reasonable recovery for second- and third-order interactions. In addition to our visual comparison, have also performed quantitative analysis the experimental results using RMSE error with the ground truth. Results are shown in Table FastCCA has shown to have better performance for First-Order interactions. However, for second- and third-order SNR results for FastICA is unable to recover a reasonable signal because the noise is more dominant. IGBSS has shown superior performance and is able recover the signal for second- and third-order interactions with better scores for both RMSE and SNR.
4 CONCLUSION
We have proposed a novel blind source separation (BSS) method, called Information Geometric Blind Source Separation (IGBSS). We have formulated our approach using the log-linear model, which enables us to introduce a hierarchical structure into its sample space to achieve BSS. We have theoretically shown that IGBSS has desirable properties for BSS such as unique recover of source signals as it solves the convex optimization problem by minimizing the KL divergence from mixed signals to source signals. We have experimentally shown that IGBSS recovers images and signals closer to the ground truth than independent component analysis (ICA), dictionary learning (DL) and non-negative matrix factorization (NMF). Thanks to the flexibility of the hierarchical structure, IGBSS is able to separate signals with complex interactions such as higher-order interactions. Our model is superior to the other approaches because it is non-degenerate and is able to recover the sign of the signal. Since approach is flexible and requires less assumptions than alternative approaches, it can be applied to various real world applications such as medical imaging, signal processing, and image processing.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
A. Agresti. Categorical Data Analysis. Wiley, 3 edition, 2012.
S. Amari. Information geometry on hierarchy of probability distributions. IEEE Transactions on Information Theory, 47(5):1701-1711, 2001.
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10(2):251- 276, 1998.
Shun-Ichi. Amari. Information geometry and its applications: Convex function and dually flat man- ifold. In F. Nielsen (ed.), Emerging Trends in Visual Computing: I.IX Fall Colloquium, ETVC 2008, Revised Invited Papers, pp. 75-102. Springer, 2009.
Shun-Ichi. Amari. Information Geometry and Its Applications. Springer, 2016.
Anthony J Bell and Terrence J Sejnowski. An information-mmaxiiizztooi approach to blind separa- tion and blind deconvolution. Neural Computation, 7(6):1129-111 1159, 1995.
Olivier Berne, C Joblin, Y Deville. JD Smith, M Rapacioli. JP Bermard, J Thomas, W Reach, and A Abergel. Analysis of the emission of very small dust particles from spitzer spectro-imagery data using blind signal separation methods. Astronomy & Astrophysics, 469(2):575-586 2007.
Christos Boutsidis and Efstratios Gallopoulos. SVD based initialization: A head start for nonnega- tive matrix factorization. Puttern Recognition, 41(4): 1350-136,, 2008.
Jean-François Cardoso. High-order contrasts for independent component analysis. Neural Compu- tation, 11(1):17--12, 1999.
Pierre Comon. Independent component analysis, a new concept? Signal Processing, 36(3):287-314, 1994.
Marco Congedo, Cédric Gouy-Pailles and Christian Jutten. On the blind source separation of human electroencephallgrann by approximate joint diagonalization of second order statistics. Clinical Newrophysiology, (19(12) 2008.
B. A. Coull and A. Agresti. Generalized log- oog-linear models with random effects, with application to smoothing contingency tables. Statistical Modelling, 3(4):251-271, 2003. 2003..
Chris HQ Ding, Tao Li, and Michael Jordan. Convex and semi-nonnegative matrix factorizations.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(1):45-55, 2008
Gerhard Gierz, Karl Heinrich Hofmann, Klaus Keimel, Jimmie D Lawson, Michael Mislove, and
Dana S Scott. Continuous lattices and domains, volume 93. Cambridge university press, 2003.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural Networks, 13(4-5):411 1-430, 2000.
Takuya Isomura and Taro Toyoizumi. A local learning rule for independent component analysis. Scientific Reports, 6:28073, 2016.
Quoe V Le, Alexandre Karpenko, Jiquan Ngiam, and Andrew Y Ng. ICA with reconstruction cost for efficient overcomplete feature learning. In Advances in Neural Information Processing Systems 24, pp. 1017-1025, 2011.
Daniel D Lee and Sebastian Seung. Algorithms for non-negative matrix factorization. In Advances in Neural Information Processing Systems 13, pp. 556-562, 2001.
Simon Luo and Mahito Sugiyama. Bias-variance trade-off in hierarchical probabilistic models us- ing higher-order feature interactions. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pp. 4488-4495, 2019.
Kevin P Murphy. Machine Learning: A Probabilistic Perspective. MIT press, 2012.
Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research, 37(23):331-33225 1997.

---

Under review as a conference paper at ICLR 2021
Karl Pearson. LIII on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901.
Hiroshi Sawada, Nobutaka Ono, Hirokazu Kameoka, Daichi Kitamura, and Hiroshi Saruwatari. A review of blind source separation methods: two converging routes to ILRMA originating from ICA and NMF. APSIPA Transactions on Signal and Information Processing, 2019.
Matthias Scholz, Fatma Kaplan, Charles L Guy, Joachim Kopka, and Joachim Selbig. Non-linear PCA: a missing data approach. Bioinformatics, 21(20):3887-3895 2005.
Mahito Sugiyama, Hiroyuki Nakahara, and Koji Tsuda. Information decomposition structured space. In 2016 IEEE International Symposium on Information Theory, pp. 575-579, 2016.
Mahito Sugiyama, Hiroyuki Nakahara, and Koji Tsuda. Tensor balancing on statistical manifold. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceed- ings of Machine Learning Research, pp. 3270-3279, 2017.
Ricardo Vigário, Veikko Jousmäki, Matti Hämäläinen, Riita Hari, and Erkki Oja. Independent component analysis for identification of artifacts in magnetoeneeehallgraphii recordings. In Advances in Neural Information Processing Systems 10, pp. 229-235, 1998.
Huan Xu, Constantine Caramanis, and Sujay Sanghavi. Robust PCA via outlier pursuit. In Advances in Neural Information Processing Systems, pp. 2496-2504, 2010.
Hui Zou, Trevor Hastie, and Robert Tibshirani. Sparse principal component analysis. Journal of Computational and Graphical Statistics, 15(2):265 286, 2006.
A APPENDIX
A.1 PARAMETER COMPUTATION FOR EACH LAYER
In the following, we give p, 77, and the gradient for each layer, which are used in gradient descent. Received Layer (Input Layer): Probability p(x) the received layer X' is obtained as
logp(x)
E1.300 + Σ 1asa00 + +0,,
(8)
ZEZ
aEA
nu " 10s = p(z).
(9)
z'EX
We do not need to compute gradient for the received layer as there is no parameter on this layer and 0g for all x X.
Source Layer (Output Layer): Probability p(z) on the source layer for each 2 E Z is given as
l(g)(z)= = -E Σ
1a + +0, 02 1az00 + 01,
(10)
z'EZ,
aEd
7) >.=-u 1.s=P(x) + Elsep((2)) = Σ (.5))()) + p(2).
(11)
#EX
z'ez
ZEX
Thus the gradient for the source layer is given
0
DkL(pllp) n7 n =
Lese (p(x) - p(x)) + p(z).
(12)
00
TEX
Mixing Layer: Probability p(a) on this layer for each a E A is given as
logp(a)
Σ La' + 01 =0. +1,,
(13)
d'EA
1/4 = Σ (laxP(x)) + Σ (1)))(()= + Σ Ilozep(a
(14)
#EA
ZE2
d'EA
Σ la<ap(() + Σ la<sp(2) p(a).
(15)
xEN
2E2
10

---

Under review as a conference paper at ICLR 2021
The gradient of the mixing layer is given as
① (KL())l) 1/ nie = E1aso - + (o p(a). 00,
(16)
ZE
ZEZ
Parameter values 0₂ in the mixing layer represent the degree of mixing between source signals. Hence they can be used to perform feature selection and extraction. For example, if 0, = in the extreme case, the corresponding node does not have any contribution to the source mixing.
A.2 FEATURE EXTRACTION FOR A 2D POINT CLOUD EXPERIMENT
We demonstrate the effectiveness for IGBSS to identify independent components on a 2-dimensional point cloud be used for feature extraction dimensionality reduction. In our experiment, we generate a 2-dimensional point cloud using two standard Student's t-distribution with 1.3 degree of freedom and have scaled the first dimension by 1/5 and the second dimension by 1 1/10 to the point cloud, illustrated in Figure 5a Then we have randomly generated mixing matrix for our experiment to generate a mixed signal shown in Figure 5b] We run the experiment on our model IGBSS using min-max normalization as pre-processing step and compare it to PCA and ICA. We apply the reverse transformation of the min-max normalization on the recovered signal and have plotted the results in Figure From the experimental results, the same scale of the point cloud. However, the sign of the signal is not recovered as we have recovered reversed sign of the signal. PCA also recovers signals which are orthogonal to the largest variance. Therefore the axes the point cloud recovered by PCA does not align with the source signal in Figure 5a| that is, the axes do not run parallel to the x- and y- axes but instead is still in the same orientation as the mixed signal. This is not what we want as the signal is still mixed, and we would like recover the signal in the same orientation as the source signal in blind source separation. ICA aims to recover statistically independent signals that are generally considered as each other. However, the
see that PCA is able to recover
the axes with the largest variances and not necessarily orthogonal limitations of ICA is that it is unable to recover the sign and the scale of the signal. Therefore the scale of the recovered signal does not match with the source signal. our experiment, we have plotted the results with unit variance as the recovered signal is generally unnormalized in ICA. Since our experiment is synthetically generated, we are able to quantitatively measure the the error each approach by normalizing both the recovered signal and the source signal by its standard deviation then computing the root mean squared error (RMSE) and the aggnal-t-noiis ratio (SNR). The results of this is shown in Table Our proposed approach IGBSS has clear advantages, where it is able to recover the same orientation as the source signal as well as preserve the signal.
Table 3: Signal-to-Noise Ratio (SNR) and Root Mean Square Error (RMSE) between the recovered signal and the latent source signal for the 2-dimensional point cloud experiment.
Model
PCA
ICA
IGBSS
RMSE
2.011
1.445
1,421
SNR
25.997
27.431
27.503
A.3 RUNTIME ANALYSIS
In our experiment, we used learning rate of 1.0 for gradient descent. Although the time complex- ity for each iteration of natural gradient is 0(12)*+|A [|| + which is larger than ( for gradient descent, natural gradient is able to reach convergence faster because it is quadratic convergence and requires significantly less iterations compared to gradient descent, which linearly converges. Increasing the size of the input will increase the size of [52] only, while the number of parameters [2], |A| remain this same. Since the complexity of natural gradient is linear with re- spect to the size [ of the input, increasing the size of the input is unlikely to increase the runtime significantly. Our experimental analysis in Figure [ supports this analysis: our model scales lin- early for both natural gradient and gradient descent when increasing the order of interactions in our model. This is because for practical application it is unlikely that A [2]. The different between the runtime for natural gradient and gradient descent becomes larger as the order of interactions increased.
11

---

Under review as a conference paper at ICLR 2021
7.5
5.0
2.5
0.0
-2
-4
-10.0
-120
-100
-80
-60
-40
-20
20
-120
-100
-80
-60
-40
-20
(a) Source Signal
(b) Mixed Signal
10.0
7.5
7.5
True Source
True Source
5.0
5.0
2.5
0.0
2.5
2.5
-5.0
-5.0
-75
-7.5
10.0
PCA Recovery
(d) ICA Recovery
IGBSS
True
3
-2
-70
-50
-40
-30
-20
10
(e) IGBSS Recovery
Figure 5: 2-dimensional point cloud experiment.
A.4 SIGN INVERSION IN ICA
When demonstrate the problem of the sign inversion in ICA. We the same experimental set-up explained in Section 3.1 on blind source separation for affine transformation. We run the experiment on the dataset used for experimental for the first order experiment and have shown the output of several runs in FastCCA to show the problem of the sign inversion in Figure 7 For the 6 runs, can see that none of the experiments were able to obtain correct sign of the signal. This means that apply FastICA to applications where the sign of the signal is important is quite problematic.
12

---

Under review as a conference paper at ICLR 2021
104
Number iterations
Rutime [sec.]
10a
10a
Natural Gradient Gradient Descent
10a
10a
Natural Gradient
10a
10a
Gradient Descent
2000
4000
6000
8000
2000
4000
6000
8000
Number of parameters
Number of paramters
Number iterations
2000
100000
Natural Gradient
Runtime [sec.]
80000
Gradient Descent
1500
Natural Gradient Gradient Descent
60000
1000
40000
500
20000
0
2
5
Order of feature interactions
Order feature interactions
Figure 6: Experimental analysis the scalability number of parameters and higher-order features in the model for both natural gradient approach and gradient descent
(a) GT
(b) Mixed
(c) Run
(d) Run 2 (c) Run
(f) Run
(g) Run
(b) Run
Figure 7: Six different runs of FastCCA with the same experimental input experimental dataset as expl with first order interactions. The different results can demonstrate that the FastICA model is non-convex leading potential problemic results such the sign inversion.
13


Reviews:
[{"review_id": "f-HH1jyx3hZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Additional comments: I have read the authors' response and the other reviews. While my initial concerns have been mostly addressed, there are still concerns from the other reviews and I have revised my score accordingly.\n\nIn addition, I am not completely satisfied with the authors' response concerning higher-order interactions. In the end, even if you add other nodes, the operations are simply matrix operations and so you can model any higher order interactions with a single matrix. This should be clarified further.\n\n______________________________________________________________________________\nThis paper uses an information geometry-based model to perform blind source separation. The problem is definitely significant and should be of interest to conference attendees. The approach appears to overcome many of the weaknesses in existing approaches and outperforms them in selected experiments. \n\n\nPros:\nIGBSS, the proposed approach, has many advantages such as convexity and potential nonlinearity allowing it to adapt to nonlinear mixing effects. The empirical results shown here also look good.\n\nThe paper appears to be technically correct.\n\nCons:\nThe experiments seem pretty simple. Only a few experiments are performed. While the results are fairly convincing, it seems like a lot more experiments could have been performed with summary statistics reported. \n\nAt the bottom of page 3, it's stated that the partial ordering allows for higher order interactions. This isn't clear to me as it still appears that only matrix operations are being performed with this model and thus only first order interactions can be modeled. Can the authors clarify this?\n\nI would be willing to raise my score if these issues were properly addressed.\n\nOther comments:\nIn Table 1, FastICA performs best wrt RMSE in experiment 2 with third order interactions and wrt SNR in experiment 2 with second order interactions. The authors should fix this.\n\nFastICA tends to do better wrt RMSE and SNR as the order increases. Do the authors have any insight into why this is?", "extra_scores": {}, "extra_text": {}}, {"review_id": "LtnF0MNDOf-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "%% post-rebuttal %%\nThough I appreciate the efforts of the authors to clarify their methodology and assumptions in their answer, these clarifications (which I still don't fully grasp or agree with) have not been reflected in the revised version. This work still needs a significant revision and, in my opinion, cannot be accepted in its current form.\n%%%%%%%%%%%%\n\n\n0) I reviewed an earlier version of this paper. The ICLR submission is similar to the earlier one, hence the similar (but updated) review. Generally, the paper is hard to follow and can only be understood by someone well-versed in both source separation and information geometry.\n\n1) Presentation of the ideas is lacking (insufficient explanations, motivations) and I failed to understand the main contribution of the paper, i.e., how the concept of “information geometric log-linear model” can be applied to BSS. The log-linear model models a discrete probability distribution of ordered events. As such the authors introduce a specific ordering for BSS that I failed to understand. Eq. (2) plays a central role but details and motivation are lacking and it’s hard to understand why this ordering is interesting and why a log-linear model makes sense.\n\n- Exemples would be welcome in Section 2.1. What is Omega ? what is an s ? what is S ? what is the meaning of Psi ? what is the intuition behind (1) ? \n\n2) I would encourage the authors to clarify the settings in which their methods is applicable. It seems to be able to tackle undetermined methods and some forms of nonlinearity, though it's not clear why. \n- What are the assumptions about the sources and mixtures ?\n- How come your method can identify the sign of the sources ? What sort of prior information is available ?\n- The preprocessing of real-valued data (basically applying exp or some form of normalisation) is disturbing.\n\n3) The experiments consider artificial mixture of images and artificial mixtures of synthetical signals with little practical significance.\n- Can you explain more clearly why ICA or NMF fail on the simple 3x3 image separation problem ? \n- Do you have any idea of an application that could benefit from the nonlinear model in Section 3.2 ?\n\n4) The introduction of the paper could be improved as it seems to reveal some misconceptions about blind source separation (BSS). For example PCA is not a method dedicated to the separation of orthogonal sources (such a problem is actually not identifiable). It is instead often used as a pre-processing in ICA (whitening). The introduction also sustain a confusion between feature extraction and source separation. These two different tasks can be handled by similar methods (ICA, NMF) but in very different settings. In source separation, the data X contains signal in rows (e.g., the cocktail party problem). In feature extraction, the focus is on the columns of X which contains data samples (such as short-time frames of a single-channel mixture). \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CCdkkzVvjp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n \n\nThis paper formulates blind source separation as a statistical estimation problem in a log-linear model and solves the statistical problem by a modified natural gradient method. \n\n\n##########################################################################\n\nReasons for score: \n \n\nThis is a complete work. However, this paper simply presents the method and experimental results and an in-depth discussion on why the proposed method should be superior than existing ones is lacking, so I do not feel this work meets the standard of ICLR. There are some misconceptions in the arguments of this paper (see cons). \n\n\n##########################################################################Pros: \n \n\nPros: \n\n1. The idea of formulating blind source separation as statistical estimation in a log-linear model seems to be novel. \n \n\n##########################################################################\n\nCons: \n \n\n1. The novelty seems limited. The main novelty is in formulating the blind source separation problem as a statistical estimation problem. The log-linear model and natural gradient method seems to be already known to experts.  \n\n2. The presentation is highly imbalanced. While the log-linear model and natural gradient method are already known, this paper spent most of the space on stating the known results and lacks an in-depth discussion on why the proposed algorithm should be superior.   The main argument supporting the proposed algorithm is that non-convex optimization is bad and convex optimization is good. This is apparently not true in general. \n\n3. The paper argues that existing non-convex approaches are not favorable because they are sensitive to initializations. This is in general not true. The authors should show empirically that sensitivity w.r.t. initializations actually poses an issue in existing approaches, or give appropriate citations supporting their claim.  \n\n4. The paper also argues that existing approaches are not favorable because they impose strong assumptions on the problem. However, I think the log-linear model assumption in this paper is not obviously weaker than the assumptions in existing approaches. \n\n5. This paper proposes using natural gradient descent to solve the convex optimization problem, because the iteration complexity of natural gradient descent is low. However, a low iteration complexity is not very meaningful; what’s meaningful is for example the overall computation complexity to achieve an \\varepsilon-approximate solution.  \n\n6. To reduce the per-iteration complexity of natural gradient descent, this paper proposes a seemingly heuristic variant in (6) and (7) and claims the algorithm also converges to the optimum. I do not see why the claim is true.   \n\n7. The presentation can be improved. The introduction to the log-linear model and natural gradient descent is quite long but not very readable to a non-expert in the two topics. It is better to have this paper checked by a native speaker, though the number of typos and grammar mistakes is acceptable.  \n\n8. The title features “Legendre transformation,” but I do not find it in the main text.  \n\n9. Typo: The set \\mathfrak{B} as defined in Section 2.3 is not exactly the set of all probability distributions. \n \n\n##########################################################################\n\nAfter reading the rebuttal: \n \n\nI keep my score. The authors should rethink how to claim the novelty in a concrete way. The presentation needs to be improved for readers not familiar with information geometry (they should be the majority). I do not get 6) in the rebuttal. \n \n\n#########################################################################", "extra_scores": {}, "extra_text": {}}, {"review_id": "1Q-KtA0an5f", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "We find he paper inappropriate for ICLR for many reasons. We give below some elements to help the authors in increasing the relevance of their work and paper.\n\nThe title is inappropriate, as Legendre transformation is not used in this work. And the word Legendre is never used in the paper. \n\nThe introduction is too long, describing many elements that are not necessary for the paper. For example, the paragraph describing the PCA and some of its variants, as well as the orthogonality constraint. In the paper, the PCA is not considered, neither the orthogonality constraint.\n\nThe authors focus on the fact that other methods are sensitive to initialization and they may converge to local optimum. In practice, the proposed method may have similar issues, as can be seen from algorithm 1 because it requires the initialization of \\theta_s. It is not clear what is the used initial values, but it seems that it is 0 and not random because it makes the obtained results constant after 49 runs. This doesn’t mean that it is independent of the initialization. \n\nIt is easy to demonstrate that the proposed method cannot jointly verify the independence on initialization and the convexity/convergence to global optimum. To show this, we can initialize with the results obtained from FastICA. As FastICA sometimes outperforms all the other methods, this means that the proposed method should provide better results that FastICA throughout iterations, because the optimization problem is supposed to be convex and it converges to the global optimum; However, this means that the method depends on the initialization. If the proposed method is supposed to be invariant to initialization, this means that it will yield the same values given in the paper, thus not outperforming FastICA is several cases. This means that it converges to an optimum that is not a globale one, as FastICA converges to better optimum solution.\n\nExperiments are not convincing, because comparing only to too old methods, such as FastICA from 2000 and dictionary learning from 1997. The authors need to compare to more recent methods from the state of the art.\n\nThere are many spelling and grammatical errors, such as disctionary.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The focus of the submission is blind source separation (BSS). The authors propose a log-linear model based formulation to tackle the task and to relax assumption/restrictions (linear mixing, non-convex objective, ...) present in previous techniques. They use the maximum likelihood approach [Eq. (3)] with natural gradient descent for optimization, and illustrate the efficiency of the approach in two toy examples (separation of mixed images and that of sin/sign/sawtooth signals).

BSS is an important task in machine learning with various applications. As assessed by the reviewers, however, the submission is in a quite preliminary stage:
(i) Section 1 is rather long, still it lacks providing relevant context to the work.
(ii) The introduction of the main ideas/motivation, the assumptions imposed, and the explanation of the notations are missing.
(iii) The usefulness of the proposed approach is questionable; the demos focus on artificial toy examples.
More work and significant revision are needed before publication.
Decision: Reject