Unrolling PALM for Sparse Semi-Blind Source Separation
Authors: Mohammad Fahes
Christophe Kervazo
Jérôme Bobin
Florence Tupin
Venue: ICLR 2022 Poster (2022)
Keywords: Algorithm Unrolling/Unfolding
Blind Source Separation
Sparse Representations
Multi-Convex Optimization
Hyper-parameter Choice

Abstract: Sparse  Blind  Source  Separation  (BSS)  has  become  a  well  established  tool  for a  wide  range  of  applications  –  for  instance,  in  astrophysics  and  remote  sensing.  Classical sparse BSS methods, such as the Proximal Alternating Linearized Minimization (PALM) algorithm, nevertheless often suffer from a difficult hyper-parameter choice, which undermines their results.  To bypass this pitfall, we propose in this work to build on the thriving field of algorithm unfolding/unrolling. Unrolling PALM enables to leverage the data-driven knowledge stemming from realistic simulations or ground-truth data by learning both PALM hyper-parameters and variables.  In contrast to most existing unrolled algorithms, which assume a fixed known dictionary during the training and testing phases, this article further emphasizes on the ability to deal with variable mixing matrices (a.k.a.  dictionaries).  The proposed Learned PALM (LPALM) algorithm thus enables to perform semi-blind source separation, which is key to increase the generalization of the learnt model in real-world applications. We illustrate the relevance of LPALM in astrophysical multispectral imaging: the algorithm not only needs up to $10^4−10^5$ times less iterations than PALM, but also improves the separation quality, while avoiding the cumbersome hyper-parameter and initialization choice of PALM. We further show that LPALM outperforms other unrolled source separation methods in the semi-blind setting.

Published as a conference paper at ICLR 2022
UNROLLING PALM FOR SPARSE SEMI-BLIND SOURCE SEPARATION
Mohammad Fahes', Christophe Kervazo', Jérôme Bobin² & Florence Tupin'
1 LTCI, Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France
2 CEA Saclay, Gif-sur- Yvette, France
ABSTRACT
Sparse Blind Source Separation (BSS) has become a well established tool for wide range of applications for instance, in astrophysics and remote ing. Classical sparse BSS methods, such as the Proximal A Aternating Linearized Minimization (PALM) algorithm, nevertheless often suffer from a difficult hyper- parameter choice, which undermines their results. To bypass this pitfall, we pro- pose in this work to build on the thriving field of algorithm unfolding/unrolling. Unrolling PALM enables to leverage the data mata-driven knowledge stemming from realistic simulations or ground toound-truth data by learning both PALM hyperparame- ters and variables. In contrast to most existing unrolled algorithms, which assume a fixed known dictionary during the training and testing phases, this article further emphasizes the ability to deal with variable mixing matrices (a.k.a. dictionar- ies). The proposed Learned PALM (LPALM) algorithm thus enables to perform semi-blind source separation, which is key to increase the generalization of the learnt model in real-world applications. We illustrate the relevance of LPALM in astrophysical multispectral imaging: the algorithm not only needs up to 10 - 10 times fewer iterations than PALM, but also improves the separation quality, while avoiding the cumbersome hyperparameter and initialization choice of PALM. We further show that LPALM outperforms other unrolled source separation methods in the semi-blind setting.
INTRODUCTION
Blind source separation (BSS): In wide range of scientific domains, including remote sensing (Dobigeon et al., 2013) and astrophysics (Picquenot al., (019). BSS a very popular paradigm aiming at finding physical decompositions of multi-valued data (Comon & Jutten 2010). Specif- ically, the data-set at hand is assumed to have been generated by a mixture of elementary signals, called sources. Assuming a linear mixing model, this mathematically writes
X = 5" N.
The objective of BSS is then to recover the unknown mixing A E Rm and sources S E Rxx matrices from the knowledge of the data set X Rmxt only (and often despite the presence of a Gaussian noise N E T Unfortunately, it has long being emphasized that BSS is an ill- posed matrix factorization problem (Comon Jutten 2010): merely minimizing a data-fidelity 2 term ||_ - A over A and S' (with [[-I[P standing for the Frobenius norm) often leads to spurious estimates which are very different from the true underlying A and S* matrices. As such, most BSS approaches impose additional priors on the sought-after factors. Among them, sparsity has led to some of the state-of-the-an results (Zibulevsky & Pearlmutter [000]] [obin et al., 2007 2015 Comon & Jutten 2010). It amounts to tackling the following minimization problem:
arg min
I - + 2
M|SS||a sparsity constraint
+ 4AA")14"1 (E(1..) (A),
A E mmxnn S E BUXt
(2)
lidelity
oblique constraint
Sygl in is the characteristic function of set B, and A denotes the i-th column of A. The first term is data fidelity term stemming from the Gaussian-noise assumption,
where ||S||| EE Ej

---

Published as a conference paper at ICLR 2022
the second one promotes sparsity while the last one enforcing each column of A to be in the lg unit ball - enables to discard degenerated solutions (in which ||A||E + oo and ||S||e + -). In the following, will specifically focus sparse BSS. Being a difficult multi-convex problem, the optimization strategy is key to minimize (). Among the various iterative optimization algorithms that have been proposed so far, PALM (Attouch al.
2010
Bolte et al. 2014) is particularly appealing due to its mathematical guarantees (Xu & Yin,
2013 because it has been proved converge critical point of 2 An iteration of PALM for
minimizing the cost function ( consists in alternating the minimization with respect to S and A. Specifically, each iteration: i) a proximal gradient step of (2) is applied to update S while A is fixed; ii) A is updated thanks to a proximal gradient step of with S fixed. The process is then repeated until the algorithm converges, see Section 2
In the context of sparse BSS though, as highlighted in (Kervazo et al. 2020), PALM suffers from severe limitations, especially due to the high sensitivity of the estimated factors with respect to the choice of the regularization parameters. It is also emphasized in (Kervazo et al. 2020) that the solution strongly depends on the initialization. Altogether, these flaws, which are confirmed in numerical experiments, undermine the results of PALM and limit its applicability in weal-world applications. Therefore, we propose to investigate how algorithm unrolling methods can bypass these limitations. The general motivation is twofold: 1) introducing learned components within the original PALM algorithm provides a data-driven regularization of the estimated A and S factors. In particular, this can help to alleviate PALM difficult
choice. 2) Since (Kervazo) et al. 2020) emphasizes the high impact of the optimization strategy on A and S estimate quality, learning some parameters of the gradient of (2) data-fidelity term is hoped to improve the results.
Algorithm unrolling (Monga et al. 2021 was pioneered in (Gregor & Le- Cun! 201O) for sparse coding. In our context, the sparse coding problem mostly corresponds to non-blind source separation, in which A = A" is known in ( Quite similarly to PALM, such a cost function is usually minimized through an iterative proximal gradient step algorithm, such as the well known Iterative Shrinkage Thresholding Algorithm (ISTA) (Daubechies et al., 2004 Blumensath & Davies, 2008). Each ISTA iteration writes as:
Algorithm unrolling:
5(K+1) = ST
((L))
1 (K)
A (A
-A x))
L
with 5(K) the estimate of S* at iteration k. L the Lipschitz constant of the gradient of 2 data- fidelity term and ST the soft-thresholding operator [
In (Gregor& & LeCun| 2010). ISTA iterations are viewed as the layers of a recurrent neural network (RNN) with parameters A 2. W1 = A A and W₂ 1. A (where I denotes the identity matrix). Instead of being fixed, the parameters 0, Wi, W₂ are however trained, using backpropa- gation, from data set. To make the distinction clearer in the following, we will write trainable variables with underlined letters, for instance 0, W1, W2. Assuming parameters to be untied across the layers, the kth layer of LISTA is thus given by:
5(K+1)= gio
( 5(K))
((( x). k= 0, K 1- 1.
Consequently, the resulting network, named Learned ISTA (LISTA), "unrolls" ISTA and truncates it to K iterations. The appeal of unrolled algorithms is twofold. On the one hand, they yield better estimates than iterative algorithms thanks to their higher representation power (Monga et al. 1021 On the other hand, numerous empirical results (Sprechmann et al. 2015: Wang et al.) 2016 Zhang & Ghanem. 2018, Zhou et al. 2018) show that they generalize well to unseen samples, thanks to their interpretable structure.
Nevertheless, when the input distribution changes too much between the training and testing phases (e.g., A changes, which is in particular the case in BSS), they often lead to impaired results as they tend to (over-) ffit the training set distribution. To make the network more robust with respect small changes in the dictionary A", it has been proposed to introduce small perturbations 6.4 ~ (O0 max) on the elements of the dictionary A during the training (Liu & Chen 2019). Although such an attempt might be beneficial in some sparse coding problems (Zhao et al.) 2011). the dictionaries vary much more BSS because of the blind nature of the problem (and the variations might not be Gaussian). In contrast, we propose in this work to use a tailored PALM- based unrolling structure to account for the fact that A" is unknown and might largely change in the training and testing phases.
ST, (v) sign(r) (0, |r| 8), which applies component-wise.
2

---

Published as a conference paper at ICLR 2022
1.1 RELATED WORKS
In source separation, algorithm unrolling has first been investigated for non-blind problems (Cowen et al. 2019) in which A" is known beforehand. For instance, (Qian et al. 2019) mostly unrolled ISTA into the same architecture as (Gregor & LeCun 2010) in order to perform non-blind Hyper- spectral Unmixing (HSU). In the same context, (Zhou & Rodrigues 202I) unrolled the ADMM algorithm (Bioucas-Diis & Figueiredo, 2010). Recently, in an attempt to perform blind HSU, (Qian et al. 2020) also unrolled ISTA to estimate S but proposed to add an extra layer in the neural network to learn A" The drawback is thus that the estimate A is hard-coded in the network archi- tecture, which is not well suited to the application on new hyperspectral images. Upon preparing this work, we became aware of the very recent work of (Xiong et al. (02T), which proposes an un- rolled architecture in order to account for the spectral variabilities in A They use an unsupervised cost function, which is a sum over the K layers of the network of the data set reconstruction error (Xirain being the training set of size ttriins i.e. here patch of the hyperspectral image):
K
Loss-
Σ || - A 18.
(5)
21. 2lerain
R=
Such loss appealing as it does not require to know the ground truth factors A and S at training- A huge drawback is that the reconstruction based eтror has long been advocated to possibly lead to spurious estimates, which are very different from the ground truth A and S factors (Comon & Jutten, 2010 Zibulevsky & Pearlmutter) [00T): Gillis 2020). Consequently, (Nasser et al.) 20211 rather added further priors on the sought after-factors in their unrolled algorithm cost function, at the price of adding extra hyperparameters to be tuned by hand. Since they focus on unrolling the multiplicative update (MU) algorithm (see also the earlier work of (Hershey et al., 2014) in this context), their network structure is quite different from both (Xiong al. 2021 1) and the algorithm we investigate in this work.
Apart from the above articles, we would like to highlight that several works about unrolling methods for plated-to-BSS problems, such as dictionary learning (Tolooshams et al. (020) or even Magnetic Resonance Imaging (Arvinte et al.) (022), exist. However, as such problems have different purposes and challenges, it is beyond the scope of this article to review them.
1.2 MOTIVATIONS AND CONTRIBUTIONS
As stated above, current source separation state-of-the-arr lets the practitioner in an unsatisfactory situation: 1) classical Blind source separation methods based algorithms such as PALM are hin- dered by a cumbersome hyperparameter choice. Until now, unsupervised unrolling- basen methods have felt short of solving such an issue: they often either lead to spurious solutions (as numerical experiments will highlight) or still require parameters to tune by hand. 2) Non-blind source separa- tion (NBSS) intrinsically assumes A" to be known and fixed during both train and test phases, which is often unrealistic in practical experiments.
In this article, we propose merge the best of both worlds by performing semi-blind source separa- tion (SBSS). Specifically, practitioners usually have access to an imperfect knowledge of the mixing matrix A", for instance by physics-derive derived simulators they might possess (which is in particular the case in astrophysics) or by approximate ground-truths of signals/images of the same nature as the ones they expect to observe (for instance, databases of spectra in remote sensing). The proposed LeaLned-PALMM (LPALM) algorithm gives principled and easy way to leverage such an imperfect knowledge to bypass the flaws of conventional sparse BSS algorithms. Specifically, the contribu- tions of this article are:
We unroll the PALM algorithm into the LPALM network, which is specifically tailored for sparse SBSS and to cope with imperfectly known mixing matrices. In particular, in contrast to usual unrolled algorithms such as LISTA (Gregor & LeCun 2010) and LISTA-CP (Chen) al. 2018), which do not alternate between A and S updates. LPALM better accounts for the variability of the mixing matrix A' which occurs in source separation problems.
We experimentally highlight the limitations of current unsupervised unrolled BSS algo- rithms: by leveraging the information contained within astrophysics simulations, LPALM largely outperforms the networks of (Xiong et al. 2021] Qian et al., 22020; Nasser et al.) 2021 in terms of i) separation quality and ii) computation time. Compared to (Nasser
3

---

Published as a conference paper at ICLR 2022
et al. 2021). the approach is different as it does not require the mixtures to be nonnega- tive. in contrast to the MU updates they use. This in particular enables to easily leverage sparsifying transforms such wavelets.
We provide an in-depth comparative study between LPALM and PALM. Specifically, we show that LPALM i) enables to automatically infer from the training set good regulariza- tion hyper-yppaeeeee parameters and initialization; ii) accelerates the estimation time up to 10 - 10a times. Altogether, these key properties make source separation reliable in real appli- cations.
2 UNROLLING FOR SEMI-BLIND SOURCE SEPARATION
As evoked above, the PALM algorithm (Bolte al. 2014) is a classical optimization algorithm in sparse BSS which sequentially and alternatively updates the factors S and A using proximal gradient steps. More precisely, for problem ( PALM boils down to the following steps:
5(k+1) ST
((())
1
(A 5(M) - -x)
L (K)
(6)
A 111-1las1
((k)
(A1))((+1))
5(k+1) 5)5(++1)))
LA (1)
where ((A) A/Lg (k) and ( (resp. LA (k)) is usually chosen the squared spectral norm of (kk) (resp. S(K)). The proximal operator of the oblique constraint is denoted II |l It corresponds to the projection on the &g unit ball of the columns of the considered matrix. A natural architecture to unroll PALM consists in keeping similar alternating design, but different update types can be considered for S and A. In the next subsection, we thus discuss the unrolling choices did, the final LPALM algorithm being summarized in Subsection 2.2
2.1 DERIVATION OF LPALM UPDATES
Update step for S: In the framework of PALM, the Š update is equivalent to special case of an ISTA, which makes LISTA (4) the most straightforward trainable model use. key feature of LISTA is that it leverages reparameterization of a standard proximal gradient descent update rule through the trainable parameters (x(k), w:(A) and g(k). The advantage of using LISTA- like update for Š would be twofold: i) learning (kk) would enable to bypass PALM cumbersome hyperparameter choice: ii) learning (=(k) and w:(k) would allow for extra degrees of freedom to efficiently estimate S* Nevertheless, in the LISTA framework, A is assumed to be fixed across the training and test sets, which significantly deviates from the present SBSS case. This might lead to deteriorated estimates of 5*, since there is in particular every reason believe W<(k) and W be
A -e-deeendent (recall that initially LISTA reparametrization was motivated by W, = I T A and W₂ A.AT))
On the other hand, the empirical success of LISTA has motivated several works on its theoretical understanding (Moreau & Bruna 2016 Giryes al. 2018 Liu & Chen 2019 Ablin et al.. 2019)
Of particular interest to us, (Chen et al. 2018) proved an asymptotic coupling of LISTA weights.
Based on this, they derived new architecture named LISTA-CP. In contrast to LISTA, using (4) as an update, a layer of LISTA-CP is given by:
5(k+1) ST (5((k) + W(t) W (X-A* 50 1: 0,1,,..,
K
(7)
LISTA-CP explicitly makes use of A* and might thus principle better suited to deal with vari- able mixing matrices, provided that A is known at least decently estimated.
In Appendix D.T. we make a thourough experimental comparison between ISTA, LISTA, LISTA- CP, and ISTA-LLT (defined in the same section) in the presence of mixing matrix variabilities in the training and testing stages. These experiments confirm the insights described above : in a nutshell, LISTA does not accelerate the estimation of S* compared to ISTA for the same number of lay- ers/iterations, and also outperformed by LISTA-CP to a quite large extent, even when inaccurate A are used in the updates. Consequently, our final choice is to opt for a LISTA-CP structure for update in the LPALM algorithm.
4

---

Published as a conference paper at ICLR 2022
Update step for A: The update of A merely aims minimizing least-squares problem, with the constraint that each column of A must belong to the lg ball of radius 1. As such, we merely propose to an ISTA-LL (ISTA with Learned AA) update where only the step size is trained. This is also motivated by the fact that a LISTA-CP-IIE specific an image/signal size, which is undesirable in real world applications.
architecture for A-update would make the network
Applying LPALM in the testing phase - wavelet pre-processing: In addition to the two above steps, which are the core of LPALM, LPALM can be applied on pre-processed data X. For instance, in our realistic experiment of Subsection 3.2 we transformed X into the isotropic undecimated wavelet domain (Starck et al.) 2010). Specifically, the data X is decomposed into the so-called coarse scale "X (i.e. large-scale approximation) and detail scales "X (i.e. fine-scale approximation), LPALM is then applied on df Xx which is sparser than sX. As such pre-processing is customary in sparse signal processing, and can be applied without loss of generality if the chosen wavelet domain orthogonal, omitted in the algorithm summary of the next subsection; see Appendix [A] for more details.
2.2
SUMMARY OF THE LPALM ALGORITHM
The LPALM algorithm: Putting together the above updates, the LPALM algorithm then reads
Input: X, outputt Apres = Ā(K) Spred = 5(K)
Initialize: A (1/ (1/vm) mxn 5(0) (0)
for in O..., 1 5(k+1) = STMW (gth) -w(+) (A ((K) (A1)) xi)
(8)
1 = 11 1111s1 Ā(0)
LA (kj (1(1)g((++))) x)
where (0(k),W W(k), T_( Ik= are the K(m + 2) trainable parameters and the notation
(0) denotes matrix size n & filled with zeros.
On top of Subsection 2.1 discussion on S and A individual updates, using LISTA-CP-HE (resp. ISTA-LL-like) update for S (resp. A) enables to keep the alternating structure of PALM. On the contrary, when using LISTA, A" (resp. S*) appears only implicitly through the W (k) and W2 matrices, which prohibits a direct alternation. To further justify the update choices we did, we numerically compare in Appendix D.2 the performances of the LPALM algorithm with other update steps (ISTA-LLT, LISTA-CP....
(k)
LPALM training loss function: In contrast to BSS unrolling methods (Xiong et al. 2021 Nasser ctt al. 2021), the semi-bling context allows to benefit from the knowledge of 5 and A during the training phase. It is then possible define loss which is sensitive to the quality of the predicted Speed and Apred such as the averaged sum of NMSE of both outputs:
Ninie [1e 1S + 1 peod 13
Σ
LOSS LPALM
Nernin f=1 11.
11:A117
where the lower-left subscript i denotes the i-th sample of the considered (training) data set. It is important to point out that this loss does not require a fine tuning of regularization parameters as in the unsupervised algorithm of (Nasser et al., 2021). Additionally, since it is based on the predicted factors and not only the reconstruction error, the proposed LPALM will be shown in the experimental section to be more robust with respect to spurious solutions, which is essential to efficiently tackle non-convex problems. This is due to the fact that we leverage the knowledge contained within the training set, leading to some implicit regularization of the original problem. Implementation details are described in Appendix A Among other, we would like to highlight that the training is easy: in particular, no fancy choice of the learning rate is needed (we used constant value of 10 (0-4 in all our
https: http://gttttubccoc htwpttthc.oooacco///////.a
5

---

Published as a conference paper at ICLR 2022
experiments).
Remark: A major difference between our approach and the previous BSS unrolled algorithms (Qian
et al. 2020 Xiong et al. 2021; Nasser et al., 2021) is the learning process itself. The latter are all designed to be trained and tested on a single matrix X: a part of its columns {x' E Rmmx1 E
[1, N] is chosen to be the training set, and the model is then tested on the entire matrix X. This is 1 can be considered & sub-problems of the form: X S +N
due to the fact that problem
[1, f). with X E R=x1 AA' E | S IX and N' E R=x1 In contrast, LPALM takes
full matrix as a training or testing instance. The data sets are thus sets of matrices, not vectors.
3 NUMERICAL EXPERIMENTS
In this section, we assess the LPALM algorithm on astrophysical data simulations corresponding to X-ray images; see details in Appendix B The data set used in Subsection 3.J consists of 900 mixing matrices A E 05x4 and 900 matrices S R-4x000, which are split into 750 training samples and 150 testing samples. The mixtures X S + N are corrupted with white Gaussian noise N so that SNR 30 dB. The spectra (i.e. the columns AA") have large variabilities over the data set, see Figure [ To mimic mildly sparse sources, the S* R Rxxx000 matrices simulated using generalized Gaussian distribution with a shape parameter = 0.3 (which is usually a good proxy to mimic the distribution of wavelet coefficients of natural images). In Subsection 3.2 the same A" used, but the sources in the test set come from real data (Picquenot et al. 2019 see Appendix
0.5
Synchrotron
Thermal
0.4
Iron
Iron
0.3
0.2
0.1
0.0
10
20
30
40
50
60
Figure 1: Illustration of the variability of the A" E 65x4 matrices within the data set of Section 100 examples of A are depicted, each column being drawn with different color.
In the following. the metric used to assess the separation quality will be the NMSE between the output of the network and the ground truth computed either on the estimated sources Spred or the estimated mixing matrices Apras:
NMSE(S (prood 8*) 1Spedd 11
and NMSE(A A*) ||Apnd - 11
(10)
S 12
|
3.1 COMPARATIVE STUDY OF PALM AND LPALM
In this section, we specifically focus on comparing the PALM and LPALM algorithms to show that the latter enables to bypass PALM flaws. As emphasized in (Kervazo al. 2020), the main pitfalls of PALM lie in its high sensitivity to the initialization and to the regularization parameter choice. On top of that, both coupled, which makes PALM very hard to use in real-world applications. The NMSE in Subsections 3.1.1 and 3.1.2 1.2 is evaluated on the estimations S of S*.
3.1.1 LPALM OVERCOMES THE INITIALIZATION PROBLEM
In this subsection, we highlight that LPALM enables to bypass PALM sensitivity to the initialization. To do that, we launched PALM with different initializations for test sample: each time, a random
When required, the usual permutation and scaling indeterminacy is corrected, commonly done in BSS.

---

Published as a conference paper at ICLR 2022
Median NMSE of PALM over the test set, with the best À chosen from the training set, compared to that of LPALM.
Table 1:
Algorithm Median NMSE
PALM 0.0176
LPALM 0.00085
matrix from the training set was chosen as an initialization A of the algorithm. The sources were initialized as 5t() = A X, with Ai(0) the pseudo-inverse of A PALM was stopped when
consecutive estimates verify both [|Š(*+1) - 5( F < 10-7 and [|.Í(*+2) ||= < 10-7. On
the other hand, LPALM was tested with the same initialization with which it was trained: A (1//m)mmm (()) (0)
Discussion: With these settings, PALM median (over the initializations"" D NMSE is 0.0201 (best: 0.0201, worse: 0.473). Furthermore, the standard deviation of PALM NMSEs with respect to the initialization, of 0.176, is huge compared to its median. This makes PALM highly unreliable in practice. On the contrary, LPALM obtained an NMSE of 0.0010 the same sample. This means that, although launched from a very generic (and much worse than the one of PALM) initialization, LPALM managed obtain better results than the best one of PALM.
3.1.2 LPALM OVERCOMES THE DIFFICULT HYPERPARAMETE CHOICE IN PALM
To try making PALM work in a realistic setting, parameter from the training set. We then compare its results to those of LPALM.
infer both its initialization and regularization
As above, PALM is initialized with a mixing matrix (0)) taken at random from the training set, and 5X0) = A+ X, where X is a data matrix from the test set to be factorized. However, the A hyperparameter is now chosen as the estimate Spred and S over the training set. It is thus the best regularization parameter, among the 30 values we tried, that can be chosen from the training set.
minimizing the median NMSE between PALM source
Discussion: Figure 2 a) shows the results of PALM over the training set. Specifically, we that the best regularization parameter is A = 2 x 10-3, for which PALM reaches median NMSE of 8.72 10-3 in 1435 iterations. On the training set still, LPALM reaches better separation qualities than PALM, which is expected as it has more capacity to benefit from the learning. Table I displays the median results of both algorithms on the test set. Interestingly, LPALM generalizes well to unseen samples and maintains much better results than PALM, with difference of more than one order of magnitude. As such, unrolling PALM enables to better leverage the knowledge of the training set than applying PALM with a good initialization and the best À value from the training set. Furthermore, we highlight that LPALM requires about 80 times fewer iterations than PALM (for the best À value, otherwise the ratio can rise even much more), making it much more tractable for large-scale data sets: on the test set, PALM required a median number of 2011 iterations, to be compared with the 25 layers of LPALM. For the sake of exhaustiveness, we furthermore plot in Figure b)) the evolution the values of the NMSE across the layers of LPALM. It is interesting to note that in this experiment, LPALMM seems to exhibit quite monotonic improvement of its estimates over the layers, although this is not guaranteed and might not be the case for other data sets.
3.2 APPLICATION TO REALISTIC DATA
In this subsection, we compare LPALM to three unrolled BSS methods: MNNBU QQian et al. 2020) SNMF-net (Xiong et al. (02T) and DNMF (Nasser al. (02T). These unsupervised methods are all designed to be applied to single X matrix: part of the columns of X is chosen to be the training set, and the model is tested on the whole X. Concerning LPALM, the training set is created in similar way in the above section; Appendix B The methods compared
data set made of the same testing mixing matrices as in the above section, but this time realistic astrophysical
'To try to separate the impact of the initialization and the regularization syper-parameter we chose for each initialization the best À value. Please note that this would not be possible in real-life experiment, in which no ground-truth is available.
7

---

Published as a conference paper at ICLR 2022
10
NMSEE Iterations
10
NA5E
10a
HMSSE
10:7
10a
10-5
10-3
10-4
10-3
10-2
10:2
inersiiterations
(b)
Figure 2: (a) PALM median NMSE and required number of iterations w.r À over 45 samples of the train set; (b) LPALM average NMSEs per layer on both train and test sets.
sources are used; see Appendix
In addition, we had to slightly modify the data set to be able to perform fair comparisons with other unrolled architectures: indeed, MNNBU and SNMF-net use a sum-to-one constraint on columns. To use them in the proper way, therefore normalized the S* matrix before applying these methods. We furthermore point out that these unrolled networks (as well as DNMF) assume the nonnegativity and sparsity of the sources in the same domain. The data are mildly sparse in the pixel domain. In LPALM, a wavelet-based pre-processing, which is customary for SBSS, is details). Due to the non-negativity constraints for the other
performed (see Appendix A for methods, it was impossible to transform the X matrix in the wavelet domain when using them; they are applied in the pixel domain.
The results of LPALM applied to realistic data are shown in Figure 3 Qualitatively, all the four
prodicted
preditted
ground truth
bruth
03
022
0.1
6.0
10
00
08
3.0
x1
Figure 3: First row: LPALM estimate and ground truth of the four columns of the mixing matrix, second row: LPALM estimated sources, third row: ground truth sources.
estimated spectra are in very good agreement with the ground truth. Similarly, the sources are well estimated. The corresponding results of the other methods are shown in Appendix C (Figures 5 6 and 7p. Quantitatively, the comparison is performed in terms of the NMSE on the estimated mixing matrix A because the ground truth sources are not the same for LPALM and other unrolled algorithms due the column normalization. The results shown in Table The first line displays
8

---

Published as a conference paper at ICLR 2022
Table 2: Comparison of LPALM with state-of-the-art methods in terms of NMSE over A
Algorithm
MNNBU
SNMF-net
DNMF
LISTA
LPALM
NMSE( Agred A²)
0.0962
0.2700
0.0466
0.0067
0.0002
Generalization (model evaluation)
0.4785
0.3053
0.5469
0.0088
0.0009
Model retraining
0.3181
0.3073
0.0922
the NMSE for the example of Figure which clearly confirms LPALM superiority over other state- of-art methods. This is due to the fact that LPALM enables to leverage the knowledge contained within the whole training set (in particular, in the ground-truth), while the other methods only deal with single images. Furthermore, DNMF is better than MNNBU and SNMF-net: this is probably related to the fact that it uses a regularized cost function and not only the reconstruction error, which is less likely to lead to spurious solutions. We further benchmarked LPALM with the LISTA ] Although LISTA gives good results, it is largely outperformed by LPALM, which is probably due to the alternating structure of the latter, enabling better deal with large variations over the A matrices. Said differently. LPALM performs much better in the semi-supervised setting than LISTA.
We further assessed the generalization capacity of the four source separation methods. The second line of the table shows the median NMSE when the same learned models are applied on the other 149 examples of the test set. These results show that the three tested tfrte- methods do not generalize well to unseen samples. This is expected, as they are here trained on single matrix and thus cannot learn the A" variabilities over the different matrices. This therefore demonstrates that the proper way to deal with A" variabilities with these networks is to re-train them for each new matrix X: this might be impractical for real-life applications, due to the computational cost and the difficult hyperparameter setting of some of them (learning rate schedule in particular). Lastly, the third line of Table 2 shows, over 20 samples, the LPALM three competitors' results when they are retrained for each new input matrix X. Interestingly, even in this setting LPALM largely outperforms the other methods, by almost two orders of magnitude. Therefore, not only LPALM does not need retraining for each new image (as it is not trained on single image), but it furthermore gives much better results than the other methods needing a retraining.
CONCLUSION
In this article, we introduce LPALM, an unrolled version of the PALM algorithm tailored for solv- ing sparse SBSS problems in the presence of variabilities of both generative factors A" and S This is key to apply unrolled sparse SBSS algorithms to real-world where the mixing matrix is generally unknown but well constrained by the underlying physical phenomena. More precisely, the chosen unrolling architecture is thoroughly justified by both methodological considerations and experimental comparisons with tate-of-the-att methods in the field. In contrast to available unrolled BSS methods that work in purely unsupervised way without knowledge of the ground truth fac- tors at training, the proposed LPALM algorithm provides significantly better solutions. Moreover, we illustrate that LPALM yields very good generalization performances, enabling to bypass PALM difficult initialization and regularization hyperparameter choice. Altogether, this makes LPALM good unrolling-base based candidate to analyze real-world data. To the best of our knowledge, this the first work using unrolling techniques for astronomical data, and we believe that it would open much more perspectives to the introduction of such techniques in astronomy. Further research includes the application of LPALM to other domains, such as remote sensing.
ACKNOWLEDGMENT
We would like to thank Fabio Acero for having provided the Chandra simulations.
A few remarks are in order: 1) it was impossible to benchmark LPALM with the other considered algo- rithms such as LISTA-CP, these require inside of their updates the knowledge of A"; 2) LISTA does not explicitly estimate the mixing matrix A but rather provides an estimation š of the sources S*. We therefore resorted an ISTA algorithm to obtain À from 5 and X.
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Pierre Ablin, Thomas Moreau, Mathurin Massias, and Alexandre Gramfort. Learning step sizes for unfolded sparse coding. arXiv preprint arXiv: 11071, 2019.
Marius Arvinte, Sriram Vishwanath, Ahmed H Tewfik, and Jonathan Tamir. Deep j-sense: Accel- erated mri reconstruction via unrolled alternating optimization. arXiv preprini arXiv:2103.0200
2021.
Hédy Attouch, Jéröme Bolte, Patrick Redont, and Antoine Soubeyran. Proximal alternating mini- mization and projection methods for nonconvex problems: An approach based on the kurdyka- tojasiewicz inequality. Mathematics of operations research, 35(2):438 (112))43-457.220 2010.
José M Bioucas-Dias and Mário AT Figueiredo. Alterntting direction algorithms for constrained Application to hyperspectral unmixing. In 2010 2nd Workshop on Hyperspec- tral Image and Signal Processing: Evolution Remote Sensing, pp. 1.4. IEEE, 2010.
sparse regression:
Thomas Blumensath and Mike E Davies. Iterative thresholding for sparse approximations. Journal of Fourier analysis and Applications, 14(5-6):629-666 654, 2008.
Jérôme Bobin, Jean-Luc Starek, Jalal Fadili, and Yassir Moudden. Sparsity and morphological diversity in blind source separation. IEEE Transactions on Image Processing, 16(11):2662-2))) 2007.
Jerome Bobin, Jeremy Rapin, Anthony Larue, and Jean-Luc Starck. Sparsity and adaptivity for the blind separation of partially correlated sources. IEEE Transactions on Signal Processing, 63(5): 1199-1213, 2015.
Jérôme Bolte, Shoham Sabach, and Mare Teboulle. Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Mathematical Programming, 146(1):459-494, 2014.
Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Theoretical linear convergence of unfolded ista and its practical weights and thresholds. arXiv preprint arXiv: 10038, 2018.
Pierre Comon and Christian Jutten. Handbook of Blind Source Separation: Independent component analysis and applications. Academic press, 2010.
Benjamin Cowen, Apoorva Nandini Saridena, and Anna Choromanska. Lsalsa: accelerated source
separation via learned sparse coding. Machine Learning, 108(8):1307-1327, 2019.
Ingrid Daubechies, Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse problems with sparsity constraint. Communications on Pure and Applied Mathematics; A Journal Issued by the Courant Institute of Mathematical Sciences, 57(11):1413- 1457, 2004.
Nicolas Dobigeon, Jean-Yves Tourneret, Cédric Richard, José Carlos M Bermudez, Stephen McLaughlin, and Alfred O Hero. Nonlinear unmixing of hyperspectral images: Models and algorithms. IEEE Signal processing magazine, 31(1):82-94, 2013.
Nicolas Gillis. Nonnegative Matrix Factorization. SIAM, 2020.
Raja Giryes, Yonina Eldar, Alex M Bronstein, and Guillermo Sapiro. Tradeoff's between con- vergence speed and reconstruction accuracy in inverse problems. IEEE Transactions Signal Processing, 66(7):1676- 1690, 2018.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference international conference on machine learning, pp. 406, 2010.
Daniel C Heinz et al. Fully constrained least squares linear spectral mixture analysis method for material quantification in hyperspectral imagery. IEEE transactions geoscience and remote sensing, 39(3):529-544 2001.
John Hershey, Jonathan Le Roux, and Felix Weninger. Deep unfolding: Model-based inspiration of novel deep architectures. arXiv preprint arXiv: 1409.2544 2014.
10

---

Published as a conference paper at ICLR 2022
Christophe Kervazo, Jérôme Bobin, Cécile Chenot, and Florent Sureau. Use of palm for él sparse matrix factorization: Difficulty and rationalization of a two-step approach. Digital Signal Pro- cessing, 97:102611, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 111iv:1412.6980, 2014.
Jialin Liu and Xiaohan Chen. Alista: Analytic weights are good as learned weights in lista. In International Conference on Learning Representations (ICLR), 2019.
Vishal Monga, Yuelong Li, and Yonina Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine, 38(2):18-44 2021.
Thomas Moreau and Joan Bruna. Understanding trainable sparse coding via matrix factorization. arXiv preprint arXiv: 00280, 2016.
José MP Nascimento and José MB Dias. Vertex component analysis: A fast algorithm to unmix hyperspectral data. IEEE transactions on Geoscience and Remote Sensing, 43(4))898-910, 2005.
Rami Nasser, Yonina Eldar, and Roded Sharan. deep unfolding for non-negative matrix factoriza- tion with application to mutational signature analysis. arXiv preprint arXiv:2108.099 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 0026-8037, 2019.
Adrien Picquenot, Fabio Acero, Jérôme Bobin, Pierre Maggi, Jean Ballet, and Gabriel W Pratt. Novel method for component separation of extended sources in x-ray astronomy. Astronomy & Astrophysics, 627:A139, 2019.
Qipeng Qian, Fengchao Xiong, and Jun Zhou. Deep unfolded iterative for hyperspectral unmixing. IGARSS 2019-2019 IEFE International Geoscience and Remote Sensing Symposium, pp. 2151-2154. IEEE, 2019.
model
Yuntao Qian, Fengchao Xiong, Qipeng Qian, and Jun Zhou. Spectral mixture model inspired net- work architectures for hyperspectral unmixing. IEEE Transactions on Geoscience and Remote Sensing. 58(10):7418-744 2020.
Pablo Sprechmann, Alexander M Bronstein, and Guillermo Sapiro. Learning efficient sparse and low rank models. IEEE transactions on pattern analysis and machine intelligence, 37(9): 18211 1833, 2015.
Jean-Lue Starck, Fionn Murtagh, and Jalal M Fadili. Sparse image and signal processing: wavelets, curvelets, morphological diversity. Cambridge university press, 2010.
Baharch Tolooshams, Sourav Dey, and Demba Ba. Deep residual autoencoders for expectation maximizationninspirred dictionary learning. IEEE Transactions on Neural Networks and Learning Systems, 32(6):2415-244 2429, 2020.
Zhangyang Wang, Qing Ling, and Thomas Huang. Learning deep 10 encoders. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Fengchao Xiong, Jun Zhou, Shuyin Tao. Jianfeng Lu, and Yuntao Qian. Snmf-net: Learning a deep alternating neural network for hyperspectral unmixing. IEEE Transactions on Geoscience and Remote Sensing, 2021.
Yangyang Xu and Wotao Yin. A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM Journal imaging sciences, 6(3):1758-1733 2013.
Jian Zhang and Bernard Ghanem. Ista-net: Interpretable aptimizationninssiree deep network for image compressive sensing. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1828-1837 2018.
11

---

Published as a conference paper at ICLR 2022
Bin Zhao, Li Fei-Fei, and Eric P Xing. Online detection of unusual events in videos via dynamic sparse coding. CVPR 2011, pp. 3313-3320. IEEE, 2011.
Chao Zhou and Miguel RD Rodrigues. An admm based network for hyperspectral unmixing tasks. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 1870-1874. IEEE, 2021.
Joey Tianyi Zhou, Kai Di, Jiawei Du, Xi Peng, Hao Yang, Sinno Jialin Pan, Ivor Tsang, Yong Liu, Zheng Qin, and Rick Siow Mong Goh. Sc2net: Sparse Istms for sparse coding. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Michael Zibulevsky and Barak A Pearlmutter. Blind source separation sparse decomposition in a signal dictionary. Neural computation, 13(4):863-88) 882, 2001.
12

---

Published as a conference paper at ICLR 2022
APPENDIX
A IMPLEMENTATION DETAILS
For the training, Adam optimizer (Kingma & Ba, 2014) is used with 31 0.9 and 32 0.999. For the implementation, Pytorch library (Paszke et al. 2019) is used.
The training is done 100 epochs with a learning rate LR=0.0001. The batch size is 1. We note that no fancy training needed (in contrast to some the other unrolled architectures we compared LPALM to).
Non-Blind Setting: The number of layers K 25. The parameters are initialized as follows:
LISTA and LISTA-CP: 1) An A is selected from the training set, 2) Ls is taken as the largest eigenvalue of __A A, 3) gk(0)
A= 10 4) (0)) 1 7,A A (only for
LISTA), 5) W AT. 5(0) 2x0--
ISTA-LLT: 1) An A is selected from the training set, 2) L3 is taken as the largest eigen- value of A" 3) (k(o)
X 10-5, 5(0) (0)2x500-
The number layers is = 25. The parameters are initialized as follows: 1) 2 matrices A and S are selected from the training set, 2) L. k(()) is taken as the largest eigenvalue of SST, for 1
LPALM:
all & [0, K 1]. 3) gh(0) 2. for all & [0, K 1]. where À 10 -5, and Ls is taken as the largest eigenvalue of A A, 4) Wk(o) Lg for all E [0, K - 11, 5) $(0) = (0) 6)
Á (0) (())
In addition to the above parameters, we applied in the test phase of Section 3.2 experiment a wavelet pre- precessccssing the data X. Let us denote by & the transform from the direct to wavelet domains, Φ is its inverse, "X the so-called coarse scale (i.e. large-scale approximation) and "X the detail scales (i.e. fine-scale). Let us further denote as At the pseudo-inverse of A. Once the model is trained, it is tested the mixture X follows:
Input: X, output: Aped = ((K),Spm = 5(K)
Initialize: Ā (1/vm) mxn 5(0) (0)
Pre-processing: ((X) x=x," 2x}
# Application of the learnt LPALM model for k in O........ 1
dg(k+1) ST
ST g(z) (d5(M)
W() (A dg(k) - -dx))
((++1) IIl--las1 (Ak)
LA (K) (A(h) dz(k+1) (x) d5(A+13T)
# Recover the source coarse scale
'Š(K) A(K) "
# Transform the source back to the direct domain 5(K) = (155(()),)
B DETAILS ABOUT THE ASTROPHYSICAL DATA
In this article, the mixing matrices come from astrophysical simulations, described in (Piequenot et al. 2019), which have been derived from real astrophysical data: the Cassiopea A supernovae remnant as observed by the X-ray space telescope Chandra chandra. harvard. edu All the sources S used are normalized.
13

---

Published as a conference paper at ICLR 2022
SYNTHETIC DATA USED IN SECTION 3.1
The A" matrices are composed of different spectra of size 65 each: i) synchrotron emission spectrum, ii) a thermal emission spectrum that is composed of various emission lines, which variabilities are similar and iii) two line emission spectra that are related to a single atomic component (e.g. iron) but with different redshifts due to the Doppler effect. The columns of the mixing matrices and their variabilities are displayed in Figure
REALISTIC DATA USED IN SECTION
The kind of sources we focused on are remnants of supernovae; unfortunately, only few samples with good estimate of the "ground truth" sources are actually available. We only have access to single hyperspectral image of such an astrophysical object, which we use for testing in Section 3.2 For the whole data set, the A" matrices are the same as in Section 3.1 Training and test set however differ concerning the sources:
Test set: In contrast Section 3.1 we here assess LPALM when S corresponds real images of supemovae remnant. Specifically, in order to be able to evaluate the different algorithm estimates, the data set X generated from ground-truth A" and S* matrices as follows:
-The A" matrices are the same as in section 3.1
-Te ground-truth sources S are obtained from real data set by the Generalized Morphological Component Analysis algorithm (GMCA -Bobin et al. (2007)) which currently state-of-art in sparse source separation.
In addition, dealing with these data is rather challenging since these sources exhibit partial correlations (pixels where more than one source is active with large amplitude), which largely hampers the performances of most BSS methods (Bobin et al., 2015). To limit the impact of multiple active sources in X, we apply a randomization on the pixels of each source. Properly tackling the partial correlation problem using unrolled architectures is left for future research, but we still present LPALM results when such a randomization is not performed Appendix D.3
Train set: in the absence of large amount of training data for the sources S*, generated synthetic sources. More specifically, we generated the fine scales of the sources according to generalized Gaussian distribution (GGD) with parameters m, and 3 (resp. mean, standard deviation and shape parameter). These parameters are fitted to the tp non-zero elements of each true reference source (t 3000 is the total number of pixels and p is the proportion of non-zero elements). Then a realization of tp samples the estimated parameters is simulated, the resulting vector concatenated with (1 p)t zero elements. Finally, random shuffling is applied to the vector. The GGD parameters are estimated from the histograms of non-zero elements of each source. This allows simulate random sources that have approximately the same statistics as the original astrophysical sources in the wavelet domain. Such simple process already enables to obtain a training set that is decent enough to enable LPALM to have good results in the test phase.
a GGD with
For both training and test sets, the data generation of was completed by adding a white Gaussian noise N such that SNR 30 dB.
C DETAILS ON SECTION EXPERIMENTS
The three methods we compare LPALM to are trained on a single matrix X. The training set con- sists in all the columns of X, which follows the observation of (Xiong et al. (02I] and (Qian al.) 2020) that the more column vectors used, the better the separation quality. Several hyperparameters (learning rate, number of epochs...) were tested in order to have the best performance w.e.. the corresponding loss function.
14

---

Published as a conference paper at ICLR 2022
The VCA Nascimento & Dias 2005) and FCLS (Heinz al. 2001) algorithms respectively used to initialize the A and S matrices, following the corresponding article recommen- dations. The initialization is shown in Figure 4 We that the columns of A well extracted by the algorithm except for the synchrotron emission.
MNNBU:
A learning rate schedule is used: Ir 0.002 at the first 99 epochs, and it is divided by 1.2 each 50 epochs starting from epoch 100. The total number of epochs for training is 500 epochs.
Initialization Ground Truth
0.20
Ground Truth
0.4
0.15
0.3
0.10
0.05
0.0
10
20
30
50
60
10
20
30
40
60
(b)
Initialization Ground Truth
Initialization
0.4
Ground Truth
0.3
0.2
0.0
0.0
10
20
30
50
60
20
30
40
60
(c)
(d)
Figure VCA results when applied to the data of Figure
The VCA Nascimento & Dias 2005) and FCLS (Heinz et al. 2001 algorithms are respectively used initialize the A and S matrices, following the corresponding article recommen- dations; see Figure The model hyperparameters are kept the same as the original paper code (learning rate schedules, number of layers...). except reducing the initial learning rate which was tuned in order to have the most possible decrease of the loss function.
SNMF-net:
DNMF:
DNMF is initialized with matrices of ones (Nasser et al. 2021). The learning rate is 0.0008, the number of epochs is 1500, and we took the best regularization parameters among the ones tested in (Naseer et al. 2021). As such, DNMF might be slightly disadvantaged compared to MNNBU and SNMF-net, since these benefit from much better initialization (note that however LPALM does not benefit from a better initialization than DNMF).
Computation of the second line of Table The second line of Table shows the losses of the models learned in the first line, when these models are evaluated on the whole testing set. The loss of MNNBU is trivially calculated using the ground truths A and the predicted matrix Apred which a hard-coded weight matrix in the network. For SNMF-net, the weights of the model are applied while initializing for each example with VCA and FCLS. For DNMF, the learned weights used to predict S from X, then nonnegative least square algorithm is applied to predict A from X and DNMF output.
D ADDITIONAL EXPERIMENTS
D.1 WHICH MODEL IS EMPIRICALLY THE BEST UPDATING S IN THE PRESENCE OF VARIABILITIES OVER A'
15

---

Published as a conference paper at ICLR 2022
Predicted Truth
0.5
Predicted
Ground Truth
0.25
0.4
0.20
0.3
0.15
0.2
0.10
0.05
0.00
60
(a)
(b)
Predicted Truth
Predicted
Truth
0.2
0.1
0.1
20
(c)
(d)
Figure 5: MNNBU predicted spectra from the data set of Section 3.2
Predicted
0.5
Predicted
0.20
0.4
0.3
0.10
0.05
20
30
(a)
(b)
Ground Truth
Ground Truth
0.3
0.3
0.2
0.0
10
20
30
40
50
60
10
20
30
40
50
60
(c)
(d)
Figure 6: SNMF predicted spectra from the data set of Section
3.2
To experimentally confirm the unrolling choices did for the S-update in Section 2.1 assess the robustness of LISTA and LISTA-CP in the presence mixing matrix variabilities at the train- ing and testing stages. We also test ISTA-LLT (ISTA with Learned L.g and Threshold), which is
16

---

Published as a conference paper at ICLR 2022
0.30
Predicted
0.6
Predicted
0.25
0.5
0.15
0.3
0.10
0.2
0.00
0.0
10
20
30
40
50
60
20
30
40
50
60
(a)
(b)
Predicted
Predicted
Truth
0.3
0.2
0.0
0.0
10
20
30
40
50
60
20
30
40
60
(c)
(d)
Figure 7: DNMF predicted spectra from the data set of Section
3.2
parameterized follows:
1 L
3(0+1) ST
5((k)
A 5(K) x) (K)
(1I)
The goal of this subsection is twofold 1) in the SBSS context, A" varies between the samples: determine which of the tested unrolling methods is more robust in such setting: 2) while the updates of LISTA-CP (Eq. 7 and ISTA-LLT (Eq. IT] require in principle the perfect knowledge of A for each sample, SBSS A" is unknown in the test phase and must be iteratively estimated within LPALM. Therefore, we evaluate LISTA-CP and ISTA-LLT when A is perfectly estimated but also when it not. The total number of layers for the unrolled methods K initialization is described in the Appendix A To assess the quality of LISTA-CP and ISTA-LLT when A" not perfectly known, corrupted version the mixing matrix A + NA. with NA white Gaussian noise such that the SNR with respect to A is 10 dB, is used in LISTA-CP ( and ISTA-LLT (TT) updates. The corresponding algorithm results are denoted as LISTA-CP (SNR = 10 dB) and ILTA-LLT (SNR 10 dB) in Figure
The weights
ISTA truncated at 25 layers is used as a benchmark to compare the architectures. Its regularization minimize the average NMSE over the
parameter is λISTA 1.1 10-5 which was chosen test set.
Discussion: Firstly, LISTA gives worse reconstruction qualities than ISTA truncated 25 layers. This might be due to the fact that LISTA trainable parameters W, (R) and W2 (k) are optimized over all the varying mixing matrices A" in the training set. Precisely, recall that LISTA reparametrization motivation was set W, T AT A and W2 7 A Being A -dependent, these parameters should rather be denoted as (A4) and (A*)). When learnt from single data sample X, it is thus likely that Wi (A*) and (2(A") are also A ddddependent. When the training set is constituted of several samples in which the A are not constant, the W1 and W2 learnt over the whole training set should resemble to an aggregate of the different (1A") and (2*)). As such, in the test phase, LISTA parameters W, and W2 might not be optimal for given mixture X which from a specific new A Such phenomenon also probably explains the superiority of LISTA-CP and ISTA-LLT LISTA, since, for given X, they precisely use information about the mixing
generated
17

---

Published as a conference paper at ICLR 2022
through the presence of A" within their updates, which enhances their robustness with respect the variabilities of A".
As expected, the results of LISTA-CP and ISTA-LLT both deteriorate when noisy mixing matrices are used within their updates (LISTA-CP (SNR = 10 dB) and ISTA-LLT (SNR = 10 dB) curves). than ISTA, LISTA-CP (SNR 10
While the results of ISTA-LLT (SNR 10 dB) become dB) still gives better estimates than ISTA. The higher ability of the LISTA-CP architecture to deal with noisy A estimate might be linked to higher capacity of the network than ISTA-LLT, there is much higher number of parameters. Such property justifies the choice of a LISTA-CP like update in LPALM.
100
10-1
NMSE
LISTA
LISTA-CP
LISTA-CP (SNR=10dB)
10-2
ISTA-LLT
ISTA-LLT (SNR=10dB)
ISTA
ISTA (SNR=10dB)
5
10
15
20
25
layers/iterations
Figure 8: NMSE of several unrolled non-blind architectures, compared to ISTA truncated to 25 layers. The LISTA-CP and ISTA-LLT curves correspond to the updates (7) and (TT) where A is perfectly known. The LISTA-CP (SNR 10 dB), ISTA-LLT (SNR 10 dB) and ISTA (SNR 10 dB) curves correspond to updates using a noisy estimate of A".
D.2 TEST OF THE POSSIBLE UPDATES WITHIN LPALM
In D.I we assessed various updates strategies for In this subsection, we try different update forms within the alternating structure of I.ALLM. In this work, we considered five update strategies: LISTA, LISTA-CP, ISTA-LLT, ISTA-LL or an explicit update (using the closed form gradient (2)
data-fidelity term). However, some update forms should be ruled out when unrolling PALM:
Any use of LISTA update, either for A or S, would prohibit an alternating structure of the algorithm;
For A, ISTA-LLT does have any sense, since there is threshold to consider. LISTA- CP cannot be considered either, as it would imply a fixed number of pixels, which is im- practical in real-life experiments;
For S, both the explicit and the ISTA-LL updates require eter X by hand, which is impractical as attested in Kervazo et al. 2020).
the regularization param-
Furthermore, an explicit update of both and S corresponds to the PALM algorithm, which has already been tested in Section 3.1 The remaining possibilities data set used in Section 3.I It can be seen that LPALM (first column) largely outperforms its counterparts using ISTA-LIT updates for S. This is likely to be linked to the higher number of
assessed in Table 3 the same
18

---

Published as a conference paper at ICLR 2022
Table 3: Comparison of the different possible updates within unrolled PALM architecture. The
median NMSE over the test set is displayed. The first column corresponds to the LPALM algorithm.
S update A-update
LISTA-CP ISTA-LL
LISTA-CP
ISTA-LLT
ISTA-LLT
Explicit
ISTA-LL
Explicit
Median NMSE( Spedd )*)
0.00085
0.00097
0.7943
0.7948
parameters in the LISTA-CP updates, enabling LPALM to have higher representation power. It is interesting to note that explicit update for A leads to very similar results LPALM in this experiment. We preferred to keep within LPALM learned component for A update, which might be beneficial in experiments where A" could be more difficult learn (for instance, larger matrix A would imply larger number of elements to estimate). Furthermore, as learning the step sizes inside of the A update only encompasses K parameters learn, the additional training cost is often negligible (as well as the over- -ittinn risk).
D.3 LPALM SEPARATION RESULTS IN PRESENCE OF PARTIAL CORRELATIONS
The original astrophysical data exhibit partial correlations, which are known to dramatically hamper the performances of most SBSS methods (Bobin et al.) (2015.. For that purpose, we previously pre- sented results where sources are randomized, to avoid such partial correlations. Figure shows the results of LPALM when applied to the raw astrophysical data. These results are of reasonable quality but show leakages that directly originate from the presence of these partial correlations. Since the training set used to train LPALM does not contain such partial correlations, it cannot learn an unmixing procedure that is robust to this type of artifacts. An in-depth comparison with other unrolled algorithms in the presence of partial correlations, well properly tackling this issue within LPALM, are left for future work.
0.20
pudetel
podated
6.3
6.9
(a)
(b)
(c)
(d)
0.000
0.0004
100
200
200
0.0044
00000
(c)
gg)
(h)
0,0000
QUBKS
0U0003
200
690
0.000
300
0,0005
200
200
8907
116
20
U84
200
i
(I)
Figure 9: Prediction ground truth of: From (a) (d) the mixing matrix, from (e) to (h) the sources (the ground truth sources are shown from (i) to (l))
19

---

Published as a conference paper at ICLR 2022
D.4 IMPACT OF THE NUMBER LAYERS IN LPALM
In this subsection, we study the impact of LPALM number of layers on the final estimate quality. The NMSE of LPALM output is plotted in figure T0 function of the number of layers K. As can be seen from the plot, LPALM first better estimates the sources and the mixing matrices when the number of layer increases, which is expected as the network representation power increases. The best K, namely K 50, is very low compared to the usual number of PALM iterations; see Figure a Nevertheless, when K becomes too large, the separation quality decreases due over- fitting. In practice, choosing K values leading to over-fitting is easily avoided since the practitioner can monitor the evolution of the validation loss, as generally done.
0.016
0.014
LPALM final estimate NMSE
0.012
0.010
0.008
0.006
0.004
0.002
10
15
20
25
30
35
40
45
50
55
number of layers
Figure 10:
Evolution of the of the NMSE over the A and estimates predicted by LPALM as a function of the network number of layers K.
D.5 IMPACT OF THE DIFFERENT TERMS IN LPALM LOSS FUNCTION
For the sake completeness, we here perform an ablation study on LPALM loss function. We test four different losses. The results are reported in Table 4
Supervised S and A, which corresponds to LPALM loss function
Nuw
||eSpped
S
IeApre 11
1
11:A 113
train
i=1
Supervised on S only, which corresponds to the loss:
Ninis
[ 13
Virain
1
Supervised on A only, which corresponds to the loss:
Num
|| Ap.Ape
11
Nurain
IAAT
Unsupervised, which corresponds to using as loss data catt--iielity term:
Num
||:X
Appes SSpeeal Il?
N
||x
20

---

Published as a conference paper at ICLR 2022
Table 4: Comparison of the estimation quality yielded by LPALM with four different losses.
Loss
Unsupervised
Supervised on S 0.00115
Supervised
Supervised
on A
on S+A 0.00112
Median (MredS 55) +NMSB Apred, A*)
0.568
0.00191
This experiment highlights that using a supervised cost function leads to much better results than merely using (unsupervised) reconstruction error. Such a phenomenon, which could already be hinted by the results in Section 3.2 of MNN-BU and SNMF-net, is well known in blind source sep- aration, in which the data-fidelity term often complemented by additional regularizations (Comon & Jutten 2010). Please note that there might be better ways to perform unsupervised learning. Such an extension of LPALM is however beyond the scope of this work.
Concerning the supervised cost functions we consider, all of them mostly obtain similar results. As such, we used in this work the sum of the NMSE over and S, which might be the most general one it does not put any emphasis on either term. According to the considered experimental setting, the practitioner might nevertheless prefer another loss: for instance, it can be hinted that if one has access to good simulator for either A S* in the training phase, loss adding more emphasis on the corresponding term might lead to better results. More generally, investigating further the impact of the loss function is left for future work.
21


Reviews:
[{"review_id": "k1eyHhXrqcQ", "reviewer": "Reviewer_fMrf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- Unrolling implementation of the PALM algorithm building on existing works e.g. LISTA-CP and ISTA-LL. The authors aim at overcoming some weaknesses of previous approaches such as the need of tuning parameters, convergence  to spurious local minima, etc.\n- This is the first work that  studies unrolled formulation of BSS problem on  hyperspectral astronomical data unmixing and show promising results of the proposed L-PALM algorithm.\n\nRemarks/Weakness\n- The authors claim that the loss function they use is more robust w.r.t. spurious solutions. However, there is no empirical evidence showing that this statement is true. For that to be clear, the authors should present an experiment that would compare their approach when a)  the loss function in (5)  vs b) the proposed loss function in (9) is used. \n- In Section 3.1 the authors compare LISTA-CP approach that is adopted for updating S vs other unrolled ISTA versions. However, the experiment presented there is not quite convincing: First the number of the sources in set to 2 which is quite small and do not correspond to a realistic number in practical applications. Second, the authors present the value NMSE as it evolves for a maximum of 5 layers. This is not quite insightful to draw safe conclusions since virtually all algorithms keep decreasing at layer=5. The plot in Figure 2 should thus show NMSE values for a larger number of layers.\n- In Section 3.2.1, the authors use what they a call as \"bad\" initialization for both PALM and L-PALM concluding that the latter is robust to bad initializations. However, the use of a single \"bad\" initialization could be potentially misleading. The authors could perform experiments for different random initializations and then measure median NMSE for providing more convincing arguments.\n- The paper lacks any theoretical argument that would give an explanation of the superiority of the proposed approach relative to state-of-the-art.\n\nTypo:\n -  page 3, 1st bullet on Contributions: tailored instead of taylored\n \n\n", "summary_of_the_paper": "The authors proposed a new deep unrolling version of the proximal alternating minimization (PALM) algorithm for addressing the semi-supervised blind source separation problem.  The main contribution of the proposed approach over relevant state-of-the-art algorithms is the use of the sum of NMSE losses of the two matrix factors and the specific ISTA-type updates that are implemented at each layer of the network for updating each matrix factor. The authors provide empirical results that demonstrate the superior performance of the proposed approach over other unrolling based state-of-the-art BSS algorithms.", "summary_of_the_review": "The paper provides an unrolling version of the well-known PALM algorithm with application to blind source separation for unmixing of hyperspectral astronomical data. The authors provide empirical results that show the advantages of the proposed approach over other relevant state-of-the-art unrolling algorithms. Given the existing state-of-the-art algorithms the technical  novelty of the current work is limited. Yet, the work presented could be of practical interest when it comes to the BSS application that is studied. However, the result are not always convincing (see Weaknesses) and could be significantly improved. The paper also lacks any theoretical justification and the evaluation of the method in only through the empirical results provided.  The paper could be also better organized since I feel the authors use a lot of subsections (especially in the experimental part), which disrupt the flow of the paper. \n\n------------------------------------------\nPost-rebuttal update\n\nThe authors have addressed my comments and concerns adding further experiments and re-organizing the paper. I, therefore, raise my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e6o5U1bxA1j", "reviewer": "Reviewer_bWys", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1] I appreciate that you explore and validate the details of your LPALM parameterization (i.e. section 3.1), which seems to often get left out in unrolling works\n\nOther Suggestions:\n\n1] your related works section is good, but you may wish to consider the following papers which also address non-blind source separation with unrolling:\nCowen et al, \"LSALSA: accelerated source separation via learned sparse coding.\"\nHershey et al, \"Deep Unfolding: Model-based Inspiration of Novel Deep Architectures\"\n\n2] in section 1.2 you say that \"[NBSS] is limited by the fact that most methods assume A* to be known and fixed...\". Isn't that the definition of NBSS?\n\n3] I don't think you broke any rules, but for example in Sec 3.1, the adjustment of line-to-line margins (and font sizes?) is too obvious that you are cramming information. I suggest trying to pare down your experimental description and maybe put some stuff in the appendix (for a specific example, instead of narrating all observations, just highlight the conclusions). Then in Section 3.3 you can give readers the really important information about preprocessing. \na) I want to emphasize that this isn't just aesthetic, but when things are crammed it's harder for readers to gather the important details. I am having a hard time understanding how your competitor methods are trained on \"a part of the pixels of the image\". This seems like a completely different training paradigm that confounds your comparison.\n\n4] In Fig 3b, instead of showing the intermediate results I think it is more interesting to show different sized LPALMs (how much NMSE gain do you get from training a longer recurrent net?). I also suggest using different markers for each competitor so that people don't have to rely on similar colors.", "summary_of_the_paper": "Problem: in the context of blind source separation (BSS), the PALM algorithm is highly sensitive to regularization parameters and initialization.\nSolution: use algorithm unrolling to learn the sensitive (hyper-)parameters.\nResults: LPALM outperforms PALM, and is compared to some competitors*.\n", "summary_of_the_review": "\n1) The reason I gave a 2/4 in \"Correctness\" is that, while you do solve the sensitivity problem w.r.t. PALM's hyperparmeters, you introduce a slew of new hyperparameters associated with training. So the real question is: How does the sensitivity to those new hyperparameters compare? (I'm not saying you're incorrect, but saying it's not well-supported that the sensitivity problem is solved; you even mention that learning rate schedule was a \"difficult\" hyperparameter to set in Sec 3.3)\n2) In terms of technical novelty or significance, I think this is standard unrolling. The thing I have not seen before is incorporating priors for both A* and S* both into the loss function.\n3)  To be clear, unrolling is fully understood to give improvements over a non-learned counterpart (although it is good to rehash it in specificity in Sec 3.1,3.2, they are not sufficient for publication in ML). So when it comes to Section 3.2, the main problem I have is that it seems the other methods were trained very differently. My main takeaway is that the overall procedure gives an improvement, which is good (or even great), but in terms of the significance I have to defer to someone who knows about this application.\n\nEdit: I want to thank the authors for giving thorough feedback to each comment and adding/updating the appendix comprehensively. I am satisfied with the responses and have raised my score accordingly. I see now that the learning-rate-schedule difficulty is a comment on the other methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QtxEYPdRXEm", "reviewer": "Reviewer_rsVe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contributions are the following:\n- An extension of unrolled alternating minimization is given that preserves the structure of the variable A\n- Experiments on both synthetic and real multispectral imaging data show improved results compared to established baselines\n\nPros:\nThe formulation of LPALM is interesting and practical. By explicitly representing A in the unrolls, the structure of A is directly used and can adapt at test-time. There is also significant time savings using the unrolled approach compared to the standard AltMin of PALM.\n\nCons:\nWhile the extension of LISTA to LPALM is intellectually interesting, there are a number of other works that also investigate unrolled alternating minimization on a sparse S and a general A, for example with (convolutional) sparse coding. There are also works that extend this to deep learning based-regularization. See for example:\n[1] Tolooshams, S. Dey, and D. Ba, “Deep residual autoencoders for expectation maximization-inspired dictionary learning,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15, 2020.o\n[2] Marius Arvinte, Sriram Vishwanath, Ahmed H. Tewfik, Jonathan I. Tamir, \"Deep J-Sense: Accelerated MRI Reconstruction via Unrolled Alternating Optimization.\" MICCAI (6) 2021: 350-360\n\nNotably, in these works, the supervised loss is only applied on the S variable. Therefore, it is unclear what value/tradeoff exists between assuming a specific A (which is estimated from the data and known to be inaccurate), vs. a true semi-blind loss function.\n\nIt is also not clear how the proposed method would compare to a simpler version of learned PALM, such as \"LPALM-LLT\", which would be similar to the ISTA-LLT while still updating A through alternating minimization. Therefore, it is not clear which component is the driving force for improved results.\n\nThere is also no theoretical justification for the choice of algorithm, and no analysis of convergence.\n\nI also had difficulty understanding what data is actually real and non-simulated, vs. realistic but simulated data. I do not understand the comments in Appendix B related to the preprocessing. Perhaps the authors could make this more clear.\n\nBased on this assessment, I believe the paper is marginally below the acceptance threshold.\n\nThe justifications for this decision is the limited number of experimental results and comparisons, missing ablations (such as supervised on S alone, or unsupervised as presented in Xiong et al.\n", "summary_of_the_paper": "Edit -- score has been updated to reflect the revisions by the authors. Remaining review below is unchanged.\n\nThis paper studies the linear mixing inverse problem, also known as blind source separation, in the context of multispectral imaging. Specifically, the work considers the situation X = AS + N, where S is assumed sparse and A has some unknown structure. The authors consider unrolling an alternating minimization based on ISTA, and learning hyperparameters and weight matrices of the unrolled optimization, which they call LPALM. Improved results are shown compared to standard AltMin (called PALM), as well as simpler unrolled methods such as LISTA that do not explicitly account for the structure of A. The learning is done using a training set where A and S are decently known and can be used as a supervised loss.\n", "summary_of_the_review": "In summary, the paper proposes an extension to LISTA, called PALM, which unrolls an alternating minimization to solve a bilinear inverse problem, i.e. sparse coding. The experiments indicate that the proposed approach is an improvement over conventional alternating minimization, but more experimental work is needed as well as proper contextualization with other unrolled methods related to unrolled blind deconvolution, dictionary learning, and other bilinear inverse problems.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GYEaEYFFebc", "reviewer": "Reviewer_7bHu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The approach presented in the paper is interesting, but several important details are missing or insufficiently explained. As a result, it is hard to judge the quality of the proposed algorithm and its potential impact. First, the issues with the existing algorithms are never well explained. When it comes to the original (unlearned) PALM, the authors state that it issues include “high sensitivity of the estimated factors with respect to the choice of regularization parameters” and that “the solution depends strongly on initialization”. What do these issues look like and what is causing them? Why do we expect that introducing a learned component would improve results here? Prior to this, the authors introduce the objective function in equation (2), but do not explain it. For example, what is the role of the oblique constraint that makes up the third term in the function?\n\nThe authors go on to introduce “semi-blind” source separation. While this is an interesting idea, it is hard to understand what is meant exactly. From the later experiments, it seems that the authors rely on access to synthetic data to provide a set of potential mixing and source matrices that can be used to train the network. It is not clear, however, how the proposed method would work in the absence of such additional information. Additionally, the source matrices are often generated by simple Gaussian sampling, so it is not obvious what additional structure is retained here.\n\nWhen describing the actual algorithm in Section 2, the authors go back and forth between various proposed steps for the algorithm before settling on a specific configuration. It is again hard to follow their argument it is never spelled out in detail what the resulting algorithm looks like. This is complicated by notation, like the uppercase pi in equation (6), that is introduced but never defined.\n\nFor their numerical experiments, the authors first concentrate on the source-matrix update step. They test various algorithms for this and conclude that one of them (LISTA-CP) performs better than the others and is chosen for inclusion in the proposed LPALM algorithm. However, this is a preliminary experiment and no attempt is made to follow this up by comparing the performance of the full LPALM algorithm with the various types of source matrix update steps. The authors also make statements like “LISTA trainable parameters … tend to be optimized by marginalizing over the varying mixing matrices A* in the training set, which could have led to deteriorated results”. What does this mean in the context of the proposed algorithm? Here and elsewhere in the paper, there are several steps missing in the explanations.\n\nThe authors then go on to compare LPALM with PALM and other state-of-the-art algorithms. It is found to perform much better than the rest, but it is hard to judge these results since the setup and methods are not very well explained. There are some additional details in the appendix, but overall, it is not clear what is happening. For example, why can the other methods only be tested on a single image? What does that mean in this context? For LPALM, the training data is synthetic – how does this affect results? Furthermore, when testing on real data, how is the “ground truth” in Figure 4(a–d) obtained?\n\nThere are also spelling errors (“taylored”), grammatical errors (“less” → “fewer”, “allowed to opt”, “a further interest to use”, and so on), unclear formulations, and unfortunate formatting choices (heavy use of underlining and boldface) that make the paper hard to read. I suggest the authors look over the text carefully before resubmitting. There are also many errors in the bibliography with missing journals, dates, and incorrectly capitalized titles.\n", "summary_of_the_paper": "EDIT: I thank the reviewers for their explanations and updates to the paper and have updated my score accordingly.\n\nThis paper introduces a method for sparse semi-blind source separation, which aims to separate signals into a linear combination of mixing components. The components are given by an unknown mixing matrix and source (or coefficient) matrix. The goal is to estimate both the mixing and source matrices from data. Since this is an ill-posed problem, certain sparsity priors are used to regularize the problem. However, this is often not sufficient and additional constraints are needed.\n\nTo solve this, the authors introduce an unrolled version of the proximal alternating linearized minimization (PALM) algorithm, where various matrices and parameters in the algorithm are learned through backpropagation in a differentiable programming framework. Some numerical results on synthetic and experimental data are provided, illustrating the potential of the approach.\n", "summary_of_the_review": "The paper proposes an interesting method with promising results, but there are several problems in the presentation of the algorithm, its motivation, and the numerical results. The main issue is a lack of clarity, which undermines the results presented. As such, I do not recommend that this paper be accepted for publication.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper develops an unrolled version of the PALM algorithm for sparse blind (or semi-blind) source separation. The unrolled version includes a soft-thresholding update, in which the thresholding parameter and one of the weight matrices is learned from data, with a least squares dictionary update, in which the step size is learned from data. The paper provides experimental results showing that this LPALM algorithm is less sensitive to the choice of hyper parameters (since step sizes, etc. are learned from data), and to the choice of the initial dictionary (perhaps since the W matrices are learned from similar examples). It also improves over PALM on experimental data from an astronomy problem. 

Reviewers expressed appreciation for the paper’s experimental results, and detailed investigation of the parameterization of unrolled PALM. They also highlighted some issues in the initial submission's exposition -- in particular, the setting of the problem (what kind of training data is available, what is the relationship between the mixing matrices A at training and at test time), and a clearer explanation of why it makes sense to learn fixed matrices W^{(k)} which do not depend on A (given that A may change at test time). The revision improved the clarity of the paper, addressing most of these concerns. The submission contributes to the discussion on how to unroll dictionary learning / blind source separation algorithms, how the unrolled algorithm should be parameterized, and demonstrates good results on multispectral data analysis.
Decision: Accept (Poster)