Title: Understanding Trainable Sparse Coding with Matrix Factorization
Authors: Thomas Moreau
Joan Bruna
Venue: ICLR 2017
Keywords: Theory
Deep learning
Optimization

Abstract: Sparse coding is a core building block in many data analysis and machine learning pipelines. Typically it is solved by relying on generic optimization techniques, such as the Iterative Soft Thresholding Algorithm and its accelerated version (ISTA, FISTA). These methods are optimal in the class of first-order methods for non-smooth, convex functions. However, they do not exploit the particular structure of the problem at hand nor the input data distribution. An acceleration using neural networks, coined LISTA, was proposed in \cite{Gregor10}, which showed empirically that one could achieve high quality estimates with few iterations by modifying the parameters of the proximal splitting appropriately.

In this paper we study the reasons for such acceleration. Our mathematical analysis reveals that it is related to a specific matrix factorization of the Gram kernel of the dictionary, which attempts to nearly diagonalise the kernel with a basis that produces a small perturbation of the $\ell_1$ ball. When this factorization succeeds, we prove that the resulting splitting algorithm enjoys an improved convergence bound with respect to the non-adaptive version. Moreover, our analysis also shows that conditions for acceleration occur mostly at the beginning of the iterative process, consistent with numerical experiments. We further validate our analysis by showing that on dictionaries where this factorization does not exist, adaptive acceleration fails.

TL;DR: We analyse the mechanisms which permit to accelerate sparse coding resolution using the problem structure, as it is the case in LISTA.

Review discussion:
The plots suggest that the proposed FacNet is almost always outperformed by LISTA and L-FISTA. This is surprising since the paper seem to suggest that LISTA is in some sense a specialization of FacNet. Any comment about this?
Thank you for your question. FacNet shares the same network structure as LISTA, but with additional constraints on the parameters.
Thus, LISTA is a generalization of FacNet, as the parameter space is larger for LISTA than for FacNet. This is coherent with the results observed in numerical experiments as LISTA outperforms FacNet, up to some optimization errors. I will update the paragraph presenting Factorization Network (end of p.6) to clarify this point.
This paper proposes a method for neural sparse coding inspired by LISTA (Gregor and LeCun 2010). A theoretical analysis is presented that attempts to explain the non-asymptotic acceleration property of LISTA (via Theorem 2.2. and Corollary 2.3).
FacNet is a specialization of LISTA, sharing the same network architecture but with additional constraints on the parameters. In numerical experiments, LISTA outperforms FacNet, up to some optimization errors. It is not clear what is the advantage of using FacNet instead of LISTA.
Overall, the paper lacks clarity in several parts. It would be good to state beforehand what the main contribution is. As stated in the clarification question/answer below, this paper would benefit from a more clear explanation about the connection of FacNet with LISTA.
Minor comments/typos:
- p. 6: "memory taps" -> tapes?
- sec 3.2: "a gap appears has the number of iterations increases" -> as?
- sec. 4: "numerical experiments of 3" -> of sec 3
Thank you for your review and the time spent on our paper.
We would like to clarify our main contribution (and we will make sure the introduction states it even more clearly): it is not to provide an alternative algorithm/model to LISTA, but rather to provide a theoretical framework that explains when and why Lista works better than non-adaptive methods.
In that respect, Facnet is a re-parametrization of LISTA that directly maps to our mathematical analysis (the factorization with matrices A and S).
Despite being a model that uses half of the effective parameter size, our experiments show that it nearly matches the performance of LISTA in all regimes, and it provides evidence that our factorization is sufficient and necessary for LISTA to succeed. The only practical interest of Facnet is that it uses less parameters than LISTA.
But LISTA will always have at least the same performance as the parameters of FacNet can be mapped directly to parameters for LISTA without loss of performances (the converse is not true). FacNet is thus a tool that permits to show numerically the link between our analysis and LISTA.
I hope that clarifies your concerns. Please let us know if there are still aspects that require clarification.
Best,
Joan
This work presents an analysis of LISTA, which originally proposes to accelerate sparse coding algorithms with some prior on the structure of the problem. The authors here propose a solid analysis of the acceleration performance of LISTA, using a specific matrix factorisation of the dictionary.
The analysis is well structured, and provides interesting insights. It would have been good to tie more closely these insights to specific properties of data or input distributions.
The learned dictionary results in Section 3.3 are not very clear: is the dictionary learned with a sort of alternating minimisation strategy that would include LISTA as sparse coding step? Or is it only the sparse coding that is studied, with a dictionary that has been learned a priori?
Overall, the paper does not propose a new algorithm and representation, but provides key insights on a well-known and interesting acceleration method on sparse coding. This is quite a nice work. The title seems however a bit confusing as 'neural sparse coding' is actually rather 'LISTA', or 'neural network acceleration of sparse coding' - basically, it is not immediate to understand what 'neural sparse coding' means...
Thank you for your feedback.
In Section 3.3, the MNIST experiment is conducted with a dictionary learned a priori, using classical dictionary learning and sparse coding techniques. The idea is to evaluate the capabilities of the models to approximate sparse codes with unconstrained  dictionaries, with potentially high coherence. We also clarified this point in the most recent manuscript version.
Regarding the title of the article, we agree that it could be confusing. There is no set name for these type of architectures besides the seminal LISTA network. We have updated the title to “Understanding Trainable Sparse Coding with Matrix Factorization”.
This paper performs theoretical analysis to understand how sparse coding could be accelerated by neural networks. The neural networks are generated by unfolding the ISTA/FISTA iterations. Based on the results, the authors proposed a reparametrization approach for the neural network architecture to enforce the factorization property and recovered the original gain of LISTA, which justified the theoretical analysis. My comments are listed below.
It is not clear about the purpose of Section 2.3.2. Adapting the factorization to the input distribution based on (15) would be time consuming because the overhead of solving (15) may not save the total time.
In fact, the approach does not use (15) but back propagation to learn the factorization parameters.
Minor comments:
- E(z_k) in (3) and (4) are not defined.
- E_x in (19) is not defined.
- Forward referencing (“Equation (20) defines…”) in the paragraph above Theorem 2.2. needs to be corrected.
Thank you for your review and for pointing out these typos.
The goal of Section 2.3.2 is to state that our theoretical analysis does not take into account the structure of the input distribution. Our theory is based on uniform bounds, for the whole input space, but the upper bounds can be improved by considering a restricted input space, i.e. using hypothesis on the inputs.
The training using SGD and back propagation is de facto using the input distribution and does not optimize uniformly.
Thus, the network parameters are linked to a factorization using (15). The implications of using this technique for different distributions are not studied in this paper, as we focused on the effect of the dictionary structure, but should be studied in a follow up paper.

Meta-review:
role: PC
data: comment: The work is fairly unique in that it provides a theoretical explanation for an empirical phenomenon in the world of sparse coding. The reviewers were overall favourable, although some reviewers thought parts of the paper were unclear or had confusion about the relationship to LISTA. I suspect the analysis here could also shed light on other problems.

Decision: Accept (Poster)