Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper proposes a network architecture composed of three interpretable components followed by a simple MLP classifier. It first applies a scattering transform followed by a learned linear projection (to reduce dimensionality). A sparse representation of these coefficients is then obtained using dictionary learning. The projection, dictionary and MLP classifier are jointly trained to minimize the classification loss. Results show that the model outperforms AlexNet on the Imagenet benchmark.

The paper is well written. The main contribution of the work is to present an architecture with composed of mathematically interpretable components achieving very high empirical performance. I find that these results are very significant. 

The second contribution is to propose a dictionary learning algorithm that uses ISTC and can be trained with gradient descent.  I think that it would be interesting to add an ablation showing the change in performance by changing N (the number of iterations in the ISTC net). Also, it would make sense to run FISTA or ISTA unrolls to see if the benefits of the faster convergence also affect classification performance.

It would be good to add to Table 1 the number of trainable parameters of each variant.

I find it a bit confusing to refer to the setting in which W is learned as ALISTA, as to me ALISTA implies using analytical W. This is clear later in the text (and makes sense from a computational standpoint). Would be good to clarify it early in the text.

Finally the paper presents a proof of exponential convergence for ALISTA in the noisy case. While this is an interesting result, it is not very closely linked to the main focus of the work. 

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: 
## Summary

The paper proposes an interpretable architecture for image classification based on a scattering transform and sparse dictionary learning approach. The scattering transform acts as a pre-trained interpretable feature extractor that does not require data. A sparse dictionary on top of this representation (the scattering coefficients) is learnt to minimize the classification error. The authors cast the dictionary learning as a classical CNN learning approach and implement an efficient solution via homotopy learning (given that some assumptions are fulfilled). The scattering transform approach is not new (as the authors mention in the paper, it was published in Oyallon et al., 2019). The main novelty comes from applying a previously published dictionary learning approach (as the authors mention in the paper, it was published in Jiao et al., 2017) on top to boost the performance. As a second contribution, the authors extend the exponential convergence proof of ISTC (Jiao et al., 2017) and ALISTA (Liu et al., 2019). In the experiments, they show that the proposed architecture, despite its simplicity, outperform AlexNet in the ImageNet classification problem.
 
## Comments

The paper is well written and the exposition is clear. The main motivation of the paper is to propose an interpretable architecture with similar performance to black box deep learning architectures. To do so, the authors put together:

- A scattering transform feature extractor: Unlike I am missing something, this is exactly what was previously proposed in (Oyallon et al., 2019). 
- A dictionary learning on top: This seems to be the biggest novelty of the paper. This component allows to boost the performance of the previously proposed architecture. However, this approach has been previously explored in the literature (Mahdizadehaghdam et al. 2018), the authors just apply it on top the extracted features. The justification of the paper lies in that previous dictionary learning approaches did not scale (convergence too slow), and so the authors use a different method recently published in (Jiao et al., 2017). 

This allow the authors to apply the method to bigger datasets ImageNet, and keep the performance above AlexNet. 

Generalizing the convergence results of ALISTA and ISTC is a nice contribution. However, my main concern is with respect the novelty of the rest of the paper. The authors do not propose a substantially different approach, rather they apply the same approach (an scalable  dictionary learning method already published in Jiao et al., 2017) on top of some extracted features (scattering coefficients)  to a different datasets. The problem with accepting the paper is that changing the dataset/dictionary learning method/features to compare with, you get a different paper, and so, in my opinion, the impact of this publication is limited.
 
Also, given that the paper main point is the interpretability of the proposed method wrt to black-box deep learning methods, I think the authors should include recent references to the active field of interpretability in the deep neural network community.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The work considers a new architecture for artificial neural networks to be used in image classification tasks. The network combines several promising previous research directions into an interesting approach (to the best of my knowledge, the architecture is novel). 

In the first step, the input representation (i.e., image) is processed using deep scattering spectrum. This is an operator proposed by Mallat (2012) and it is known for extracting robust features with properties such as translation invariance and Lipschitz continuity of the operator mapping. As a result, such a representation is likely to be more stable to perturbations and noise in the inputs.
The scattering operator of the second and higher orders tends to produce a large number of coefficients. Such a representation is, in general, sparse with many of the higher order coefficients equal to zero. To lower the dimension of the scattering representation, the architecture employs a linear projection operator. For example, one can take principal component analysis and project to a sub-space retaining most of the variation present in the data.
The first two blocks (scattering and linear projection) are unsupervised and, thus, kept fixed during learning. 

The third block in the architecture aims at finding a sparse coding dictionary that will take into account instances and labels. It builds a dictionary using a convolutional neural network and finds sparse coding using previous work on dictionary learning (e.g., Donoho & Elad, 2006: Marial et al., 2011; Jiao et al., 2017 etc). The algorithm works by computing the sparse encoding vector (see Eq. 1) in the forward pass and updating the convolutional parameters as well as sparsity controlling hyperparameter in the backward step. To speed up the convergence, the authors rely on homotopy iterated thresholding illustrated in Figure 2.

The approach is evaluated on ImageNet and empirical results demonstrate that the removal of the sparse encoding block amounts to a significant performance degradation. The results also establish a minor improvement in the accuracy as a result of adding a linear projection matrix (i.e., principal component analysis applied to scattering coefficients). Overall, the network shows promising performance on ImageNet by doing better than AlexNet. The result is not yet 'competitive' with ResNets but it might be worth to pursue this direction of research in the future.

The work is properly structured with well organized materials from previous work. The clarity, however, could be improved in several places. In particular, the last paragraph in Section 3.1 does not explain the algorithm clearly. The confusing part is how to update the dictionary (i.e., convolutional network) parameters. One might infer from the current materials that first the problem in Eq. (1) is solved to find \alpha and that solution is fixed. Then, for that setting of the sparse encoding vector one will train the network parameters. Section 3.2 then givens an iterative procedure in Eq. (3) and Figure 2, which suggest that in a forward pass the dictionary representation is computed using some setting of parameters and \alpha is updated per Eq. (3). Following this, the gradient of the convolutional parameters is computed (\alpha_{n + 1} is differentiated, which means that the gradient depends on other \alpha iterates). I believe that this is the crux of the methodological/conceptual contribution and requires proper explanation with proper illustration of the forward-backward pass.

In Section 3.2 (homotopy iterated thresholding and ALISTA), there is a matrix W which comes ad-hoc. There should be some motivation and gentle introduction. At the moment, it is completely justified to ask why one needs this matrix and why the approach would not work without Proposition 3.1. Please add some discussion and make sure things are properly motivated and gently introduced. This will also place the theoretical contribution in the proper context and strengthen the work.

Just below Figure 2, I fail to follow how the number of layers N relates to the iterative algorithm? Does it mean that you would actually have blocks per each \alpha_n for some N indices (this again refers to previous comment on clarity)?

Can you please use \ell_1 or l_1 notation for sparse dictionary coding. The current symbol reads as 1 to the power of 1 and it is very confusing (never seen it before in the context of sparse encodings).
review_title: Official Blind Review #1

Meta-review: After the rebuttal period the ratings on this paper increased and it now has a strong assessment across reviewers. The AC recommends acceptance.
Decision: Accept (Poster)