Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper presents results on Dictionary Learning through l4 maximization. The authors base this paper heavily off of the formulation and algorithm in Zhai et. al. (2019) "Complete dictionary learning via l4-norm maximization over the orthogonal group". The paper draws connections between complete dictionary learning, PCA, and ICA by pointing out similarities between the objectives functions that are optimized as well as the algorithms used. The paper further presents results on dictionary learning in the presence of different types of noise (AWGN, sparse corruptions, outliers) and show that the l4 objective is robust to different types of noise. Finally the authors apply different types of noise to synthetic and real images and show that the dictionaries that they learn are robust to the noise applied.

Overall this paper makes significant contributions by extending the work in the paper referenced above to noisy dictionary learning settings and I would vote to accept based on these results.

The connections between Complete Dictionary Learning, PCA and ICA are interesting, but the algorithmic analogies seem superficial in my opinion. There are a lot of algorithms which follow a projected/proximal gradient descent scheme. If there are any deeper connections between the specific algorithms discussed, they should be spelled out more clearly. One point of clarification that I would like to raise is the similarity between the kurtosis and l4 objectives. This paper could be strengthened by delineating the conditions under which one would learn an ICA basis vs a Complete Dictionary. It seems to me that the only difference is in the generative model, and that maximizing the same objective under different data conditions could return an ICA basis or a Complete Dictionary. 

The robustness theory and experiments on synthetic data are reasonable and demonstrate that complete dictionary learning is robust to the different noise conditions. I would like to how this technique compares to other complete dictionary learning algorithms (ER-SpUD, Complete dictionary learning over the sphere - Sun, Qu, Wright 2015) and whether the l4 objective is unique in providing this robustness. Another central claim of Zhai et. al. 2019 seems to be that l4 maximization is able to recover the entire dictionary at once, vs other algorithms that recover the dictionary one column at a time. To test this, I would like to see runtime evaluations and comparisons to other algorithms. While the claim of recovering the entire dictionary is true, it seems to me that requiring an SVD at each iteration would be very expensive. I am not completely convinced that the approach of estimating the entire dictionary would indeed be faster.

To summarize, I believe this paper would be a good addition to the literature on l4 maximization algorithms for dictionary learning. I am willing to adjust my score based on responses to the above concerns.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper explores the recently proposed $\ell^4$-norm maximization approach for solving the sparse dictionary learning (SDL) problem. Unlike other previously proposed methods that recover the dictionary one row/column at a time, for an orthonormal dictionary, the $\ell^4$-norm maximization approach is known to recover the entire dictionary once for all. 

This paper shows that $\ell^4$-norm maximization has close connections with the PCA and ICA problem. Furthermore, focusing on the MSP algorithm for solving the  $\ell^4$-norm maximization formulation, the paper highlights the connections of this fixed-point style algorithm with such algorithms for PCA and ICA. Subsequently, the paper studies the behavior of the MSP algorithm in the presence of noise, outliers, and sparse corruption. Unlike PCA, surprisingly, the MSP algorithm is shown to be robust to outliers and sparse corruption.

Overall, the paper makes a nice effort towards better understanding the relatively new $\ell^4$-norm maximization approach and its connection with other well-understood problems in the literature. Moreover, the paper takes the right step by studying the effect of non-ideal signal measurements on the underlying goal of dictionary learning. That said, the reviewer feels that, in the current form, the results in the paper are not novel enough to warrant an acceptance to ICLR. The connection of the $\ell^4$-norm maximization formulation with ICA have been previously noted in other paper, so this would hardly qualify as a novel contribution. The analysis of the MSP algorithm in the presence of noise, outlier, and sparse corruption is not comprehensive enough. It would have been nice if the authors had provided a non-asymptotic analysis of the MSP algorithm in the presence of non-ideal measurements. Also, it is not clear how interesting the outlier formulation presented in the paper is. Shouldn't one consider outliers that go beyond the Gaussian distribution, ideally arbitrary outliers?
review_title: Official Blind Review #1

Meta-review: Main content:

Blind review #3 summarizes it well:

This paper presents results on Dictionary Learning through l4 maximization. The authors base this paper heavily off of the formulation and algorithm in Zhai et. al. (2019) "Complete dictionary learning via l4-norm maximization over the orthogonal group". The paper draws connections between complete dictionary learning, PCA, and ICA by pointing out similarities between the objectives functions that are optimized as well as the algorithms used. The paper further presents results on dictionary learning in the presence of different types of noise (AWGN, sparse corruptions, outliers) and show that the l4 objective is robust to different types of noise. Finally the authors apply different types of noise to synthetic and real images and show that the dictionaries that they learn are robust to the noise applied.

--

Discussion:

Reviews agree about the interesting work, including the connections of complete dictionary learning with classic PCA and ICA (after further clarification during the rebuttal period). Additional empirical strengthening during the rebuttal period also addressed a reviewer concern.

--

Recommendation and justification:

As review #3 wrote, "Overall this paper makes significant contributions by extending the work in [Zhai et. al's (2019) "Complete dictionary learning via l4-norm maximization over the orthogonal group"] to noisy dictionary learning settings".
Decision: Accept (Poster)