Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Edit -- score has been updated to reflect the revisions by the authors. Remaining review below is unchanged.

This paper studies the linear mixing inverse problem, also known as blind source separation, in the context of multispectral imaging. Specifically, the work considers the situation X = AS + N, where S is assumed sparse and A has some unknown structure. The authors consider unrolling an alternating minimization based on ISTA, and learning hyperparameters and weight matrices of the unrolled optimization, which they call LPALM. Improved results are shown compared to standard AltMin (called PALM), as well as simpler unrolled methods such as LISTA that do not explicitly account for the structure of A. The learning is done using a training set where A and S are decently known and can be used as a supervised loss.

main_review: The main contributions are the following:
- An extension of unrolled alternating minimization is given that preserves the structure of the variable A
- Experiments on both synthetic and real multispectral imaging data show improved results compared to established baselines

Pros:
The formulation of LPALM is interesting and practical. By explicitly representing A in the unrolls, the structure of A is directly used and can adapt at test-time. There is also significant time savings using the unrolled approach compared to the standard AltMin of PALM.

Cons:
While the extension of LISTA to LPALM is intellectually interesting, there are a number of other works that also investigate unrolled alternating minimization on a sparse S and a general A, for example with (convolutional) sparse coding. There are also works that extend this to deep learning based-regularization. See for example:
[1] Tolooshams, S. Dey, and D. Ba, “Deep residual autoencoders for expectation maximization-inspired dictionary learning,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15, 2020.o
[2] Marius Arvinte, Sriram Vishwanath, Ahmed H. Tewfik, Jonathan I. Tamir, "Deep J-Sense: Accelerated MRI Reconstruction via Unrolled Alternating Optimization." MICCAI (6) 2021: 350-360

Notably, in these works, the supervised loss is only applied on the S variable. Therefore, it is unclear what value/tradeoff exists between assuming a specific A (which is estimated from the data and known to be inaccurate), vs. a true semi-blind loss function.

It is also not clear how the proposed method would compare to a simpler version of learned PALM, such as "LPALM-LLT", which would be similar to the ISTA-LLT while still updating A through alternating minimization. Therefore, it is not clear which component is the driving force for improved results.

There is also no theoretical justification for the choice of algorithm, and no analysis of convergence.

I also had difficulty understanding what data is actually real and non-simulated, vs. realistic but simulated data. I do not understand the comments in Appendix B related to the preprocessing. Perhaps the authors could make this more clear.

Based on this assessment, I believe the paper is marginally below the acceptance threshold.

The justifications for this decision is the limited number of experimental results and comparisons, missing ablations (such as supervised on S alone, or unsupervised as presented in Xiong et al.

summary_of_the_review: In summary, the paper proposes an extension to LISTA, called PALM, which unrolls an alternating minimization to solve a bilinear inverse problem, i.e. sparse coding. The experiments indicate that the proposed approach is an improvement over conventional alternating minimization, but more experimental work is needed as well as proper contextualization with other unrolled methods related to unrolled blind deconvolution, dictionary learning, and other bilinear inverse problems.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: EDIT: I thank the reviewers for their explanations and updates to the paper and have updated my score accordingly.

This paper introduces a method for sparse semi-blind source separation, which aims to separate signals into a linear combination of mixing components. The components are given by an unknown mixing matrix and source (or coefficient) matrix. The goal is to estimate both the mixing and source matrices from data. Since this is an ill-posed problem, certain sparsity priors are used to regularize the problem. However, this is often not sufficient and additional constraints are needed.

To solve this, the authors introduce an unrolled version of the proximal alternating linearized minimization (PALM) algorithm, where various matrices and parameters in the algorithm are learned through backpropagation in a differentiable programming framework. Some numerical results on synthetic and experimental data are provided, illustrating the potential of the approach.

main_review: The approach presented in the paper is interesting, but several important details are missing or insufficiently explained. As a result, it is hard to judge the quality of the proposed algorithm and its potential impact. First, the issues with the existing algorithms are never well explained. When it comes to the original (unlearned) PALM, the authors state that it issues include “high sensitivity of the estimated factors with respect to the choice of regularization parameters” and that “the solution depends strongly on initialization”. What do these issues look like and what is causing them? Why do we expect that introducing a learned component would improve results here? Prior to this, the authors introduce the objective function in equation (2), but do not explain it. For example, what is the role of the oblique constraint that makes up the third term in the function?

The authors go on to introduce “semi-blind” source separation. While this is an interesting idea, it is hard to understand what is meant exactly. From the later experiments, it seems that the authors rely on access to synthetic data to provide a set of potential mixing and source matrices that can be used to train the network. It is not clear, however, how the proposed method would work in the absence of such additional information. Additionally, the source matrices are often generated by simple Gaussian sampling, so it is not obvious what additional structure is retained here.

When describing the actual algorithm in Section 2, the authors go back and forth between various proposed steps for the algorithm before settling on a specific configuration. It is again hard to follow their argument it is never spelled out in detail what the resulting algorithm looks like. This is complicated by notation, like the uppercase pi in equation (6), that is introduced but never defined.

For their numerical experiments, the authors first concentrate on the source-matrix update step. They test various algorithms for this and conclude that one of them (LISTA-CP) performs better than the others and is chosen for inclusion in the proposed LPALM algorithm. However, this is a preliminary experiment and no attempt is made to follow this up by comparing the performance of the full LPALM algorithm with the various types of source matrix update steps. The authors also make statements like “LISTA trainable parameters … tend to be optimized by marginalizing over the varying mixing matrices A* in the training set, which could have led to deteriorated results”. What does this mean in the context of the proposed algorithm? Here and elsewhere in the paper, there are several steps missing in the explanations.

The authors then go on to compare LPALM with PALM and other state-of-the-art algorithms. It is found to perform much better than the rest, but it is hard to judge these results since the setup and methods are not very well explained. There are some additional details in the appendix, but overall, it is not clear what is happening. For example, why can the other methods only be tested on a single image? What does that mean in this context? For LPALM, the training data is synthetic – how does this affect results? Furthermore, when testing on real data, how is the “ground truth” in Figure 4(a–d) obtained?

There are also spelling errors (“taylored”), grammatical errors (“less” → “fewer”, “allowed to opt”, “a further interest to use”, and so on), unclear formulations, and unfortunate formatting choices (heavy use of underlining and boldface) that make the paper hard to read. I suggest the authors look over the text carefully before resubmitting. There are also many errors in the bibliography with missing journals, dates, and incorrectly capitalized titles.

summary_of_the_review: The paper proposes an interesting method with promising results, but there are several problems in the presentation of the algorithm, its motivation, and the numerical results. The main issue is a lack of clarity, which undermines the results presented. As such, I do not recommend that this paper be accepted for publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Problem: in the context of blind source separation (BSS), the PALM algorithm is highly sensitive to regularization parameters and initialization.
Solution: use algorithm unrolling to learn the sensitive (hyper-)parameters.
Results: LPALM outperforms PALM, and is compared to some competitors*.

main_review: Strengths:
1] I appreciate that you explore and validate the details of your LPALM parameterization (i.e. section 3.1), which seems to often get left out in unrolling works

Other Suggestions:

1] your related works section is good, but you may wish to consider the following papers which also address non-blind source separation with unrolling:
Cowen et al, "LSALSA: accelerated source separation via learned sparse coding."
Hershey et al, "Deep Unfolding: Model-based Inspiration of Novel Deep Architectures"

2] in section 1.2 you say that "[NBSS] is limited by the fact that most methods assume A* to be known and fixed...". Isn't that the definition of NBSS?

3] I don't think you broke any rules, but for example in Sec 3.1, the adjustment of line-to-line margins (and font sizes?) is too obvious that you are cramming information. I suggest trying to pare down your experimental description and maybe put some stuff in the appendix (for a specific example, instead of narrating all observations, just highlight the conclusions). Then in Section 3.3 you can give readers the really important information about preprocessing. 
a) I want to emphasize that this isn't just aesthetic, but when things are crammed it's harder for readers to gather the important details. I am having a hard time understanding how your competitor methods are trained on "a part of the pixels of the image". This seems like a completely different training paradigm that confounds your comparison.

4] In Fig 3b, instead of showing the intermediate results I think it is more interesting to show different sized LPALMs (how much NMSE gain do you get from training a longer recurrent net?). I also suggest using different markers for each competitor so that people don't have to rely on similar colors.
summary_of_the_review: 
1) The reason I gave a 2/4 in "Correctness" is that, while you do solve the sensitivity problem w.r.t. PALM's hyperparmeters, you introduce a slew of new hyperparameters associated with training. So the real question is: How does the sensitivity to those new hyperparameters compare? (I'm not saying you're incorrect, but saying it's not well-supported that the sensitivity problem is solved; you even mention that learning rate schedule was a "difficult" hyperparameter to set in Sec 3.3)
2) In terms of technical novelty or significance, I think this is standard unrolling. The thing I have not seen before is incorporating priors for both A* and S* both into the loss function.
3)  To be clear, unrolling is fully understood to give improvements over a non-learned counterpart (although it is good to rehash it in specificity in Sec 3.1,3.2, they are not sufficient for publication in ML). So when it comes to Section 3.2, the main problem I have is that it seems the other methods were trained very differently. My main takeaway is that the overall procedure gives an improvement, which is good (or even great), but in terms of the significance I have to defer to someone who knows about this application.

Edit: I want to thank the authors for giving thorough feedback to each comment and adding/updating the appendix comprehensively. I am satisfied with the responses and have raised my score accordingly. I see now that the learning-rate-schedule difficulty is a comment on the other methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors proposed a new deep unrolling version of the proximal alternating minimization (PALM) algorithm for addressing the semi-supervised blind source separation problem.  The main contribution of the proposed approach over relevant state-of-the-art algorithms is the use of the sum of NMSE losses of the two matrix factors and the specific ISTA-type updates that are implemented at each layer of the network for updating each matrix factor. The authors provide empirical results that demonstrate the superior performance of the proposed approach over other unrolling based state-of-the-art BSS algorithms.
main_review: Strengths

- Unrolling implementation of the PALM algorithm building on existing works e.g. LISTA-CP and ISTA-LL. The authors aim at overcoming some weaknesses of previous approaches such as the need of tuning parameters, convergence  to spurious local minima, etc.
- This is the first work that  studies unrolled formulation of BSS problem on  hyperspectral astronomical data unmixing and show promising results of the proposed L-PALM algorithm.

Remarks/Weakness
- The authors claim that the loss function they use is more robust w.r.t. spurious solutions. However, there is no empirical evidence showing that this statement is true. For that to be clear, the authors should present an experiment that would compare their approach when a)  the loss function in (5)  vs b) the proposed loss function in (9) is used. 
- In Section 3.1 the authors compare LISTA-CP approach that is adopted for updating S vs other unrolled ISTA versions. However, the experiment presented there is not quite convincing: First the number of the sources in set to 2 which is quite small and do not correspond to a realistic number in practical applications. Second, the authors present the value NMSE as it evolves for a maximum of 5 layers. This is not quite insightful to draw safe conclusions since virtually all algorithms keep decreasing at layer=5. The plot in Figure 2 should thus show NMSE values for a larger number of layers.
- In Section 3.2.1, the authors use what they a call as "bad" initialization for both PALM and L-PALM concluding that the latter is robust to bad initializations. However, the use of a single "bad" initialization could be potentially misleading. The authors could perform experiments for different random initializations and then measure median NMSE for providing more convincing arguments.
- The paper lacks any theoretical argument that would give an explanation of the superiority of the proposed approach relative to state-of-the-art.

Typo:
 -  page 3, 1st bullet on Contributions: tailored instead of taylored
 


summary_of_the_review: The paper provides an unrolling version of the well-known PALM algorithm with application to blind source separation for unmixing of hyperspectral astronomical data. The authors provide empirical results that show the advantages of the proposed approach over other relevant state-of-the-art unrolling algorithms. Given the existing state-of-the-art algorithms the technical  novelty of the current work is limited. Yet, the work presented could be of practical interest when it comes to the BSS application that is studied. However, the result are not always convincing (see Weaknesses) and could be significantly improved. The paper also lacks any theoretical justification and the evaluation of the method in only through the empirical results provided.  The paper could be also better organized since I feel the authors use a lot of subsections (especially in the experimental part), which disrupt the flow of the paper. 

------------------------------------------
Post-rebuttal update

The authors have addressed my comments and concerns adding further experiments and re-organizing the paper. I, therefore, raise my score.
review_title: null

Meta-review: The paper develops an unrolled version of the PALM algorithm for sparse blind (or semi-blind) source separation. The unrolled version includes a soft-thresholding update, in which the thresholding parameter and one of the weight matrices is learned from data, with a least squares dictionary update, in which the step size is learned from data. The paper provides experimental results showing that this LPALM algorithm is less sensitive to the choice of hyper parameters (since step sizes, etc. are learned from data), and to the choice of the initial dictionary (perhaps since the W matrices are learned from similar examples). It also improves over PALM on experimental data from an astronomy problem. 

Reviewers expressed appreciation for the paper’s experimental results, and detailed investigation of the parameterization of unrolled PALM. They also highlighted some issues in the initial submission's exposition -- in particular, the setting of the problem (what kind of training data is available, what is the relationship between the mixing matrices A at training and at test time), and a clearer explanation of why it makes sense to learn fixed matrices W^{(k)} which do not depend on A (given that A may change at test time). The revision improved the clarity of the paper, addressing most of these concerns. The submission contributes to the discussion on how to unroll dictionary learning / blind source separation algorithms, how the unrolled algorithm should be parameterized, and demonstrates good results on multispectral data analysis.
Decision: Accept (Poster)