Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces structured deep network architectures
that can be analyzed mathematically and have high classification accuracies on
complex image databases. The proposed mechanism consists of iterating over tight frame contractions. 
They also show that spatial filters do not need to be learned, and can be defined from wavelet frames.

############

Overall inclined for accepting the paper although I am a bit hesitant due to the lack of experimental details and/or code. Overall, the 
 main idea is interesting and novel and the paper is well written. 

############

Pros

* Interesting idea and theoretical results/implications. 

############

Cons 
* Details on the experiments are largely missing, i.e. parameters etc are not listed anywhere in the experimental section.
* No code provided or even mentioned in the paper. This makes the experimental verification harder. 
review_title: Interesting Paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes using networks that are composed of tight frames to analyze the clustering property of networks across layers.
Yet, the main focus of the paper in the first part is to construct the tight frame-based networks and then in the second part to train scattering transforms based networks. 
While the ideas are interesting I have several concerns:

1. The idea of encouraging tight frame structure is not new and appeared already in several works in the literature.
2. The idea of training a scattering transform is not new and has been done before. For example, there is a work by Mallat that shows that one may just add 1x1 convs to the scattering transform and train it. So the current work is not of much difference. 

Given these two concerns, I don't think the current novelty is sufficient for publication. 

=========================================================================================

Updated review:
Thank you for the clarification. The previous version was indeed confusing to me. I have raised my score although I think some points still need to be addressed in the revision following my previous comments as they were not fully addressed nor in the response neither in the revision: 
1.  The concern with respect to previous works is not only regarding Parseval networks. There are other more recent works that use orthogonality constraints on the network. Such examples include
https://ieeexplore.ieee.org/document/8877742
https://openaccess.thecvf.com/content_cvpr_2017/papers/Xie_All_You_Need_CVPR_2017_paper.pdf
https://proceedings.neurips.cc/paper/2018/hash/bf424cb7b0dea050a42b9739eb261a3a-Abstract.html

All these works show a similar observation to the one claimed in the paper that by using orthogonality (or frame-like) operators one may train a network without skip connections and get similar results. 
Indeed, in the paper, more observations are being made that are different than what is presented in these works but a more proper comparison should be made. 

2. This is the work the authors should look at by Mallat
https://arxiv.org/pdf/1809.06367.pdf
They get similar performance to ResNet with a scattering transform-based network. 
Indeed, also here it is not exactly the same network that the authors here are using but there are remarkable similarities and these should be well addressed. 
review_title: The paper suggests using tight frames with neural networks but end up with training scattering transforms

final_score: rating: 7: Good paper, accept
review_content: This work introduces the concatenation of a tight frame with a scalar non-expansive operator (mainly the modulus and soft-thresholding) as a unit, which - when applied to scattering transform coefficients - can yield high quality image classification results, improve the Fisher discriminant ratio, and (for soft-thresholding) allow some mathematical analysis in the sense of variance reduction bounds.

The paper is well-written, well-motivated, interesting and, to the best of my knowledge, novel. It combines theoretical insights with practical results that are only slightly worse than ResNet-18 on ImageNet. 

On the negative side I only have a couple of minor things:
- It is slightly confusing to me that the operator $\rho$ is called "contracting" instead of "non-expansive". In the optimization literature a contraction would have to satisfy the stated property with an additional factor \gamma < 1 on the right hand side. I think it is worth rephrasing this everywhere (even in the title) to avoid confusion. 
- I think it is worth stating how well the Parseval regularization enforces the orthogonality constraint for all numerical experiments. Why does $\alpha$ vary quite strongly from experiment to experiment?
- Is the batch normalization for stabilizing the learning of W' really necessary despite the rather well-behaved remaining architecture?
- On the practical side, it would have been interesting to see if the remaining gap to ResNet can be closed if the fixed spatial wavelet filters are (partially) replaced by learnable ones. To highlight the advantages of avoiding to learn such filters, the overall number of learnable parameters or the training times could be compared. 

In summary, I think this is an interesting paper that does merit a publication. 


----------------------
After the rebuttal: I'd like to thank the authors for their answers, particularly for resolving the confusion about the term "contraction". I believe this is a good paper and stick to my rating of recommending its acceptance. 
review_title: Interesting analysis for a specific type of network

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This is an interesting paper which seeks to explain the recently observed phenomenon in deep net image classification, whereby variance of activations for a fixed class becomes small / collapses in deeper layers, while the class means remain well-separated. The authors make an intriguing connection with the classical results of Donoho & Johnstone (1994) on soft thresholding to show that for a certain mixture of Gaussian mixtures, soft thresholding reduces the order of intra-class variation from O(d) to O(log d) while only moving the means by O(log d), provided that the means of the mixture components are compressible (approximately sparse) in the given frame.


A. Strong points

- The simplified framework proposed by the authors seems to indeed distill the key aspects of the problem and the results on soft thresholding are very promising. It is reassuring to see a connection to sparsity / compressibility which plays important but still elusive role in deep network classification.
- Theorem 2.2. which uses the soft thresholding non-linearity gives an intriguing insight into how the above reduction of variance could occur. 
- The numerical experiments convincingly illustrate the theoretical discussions.
- As a meta point, I very much agree with the statement that excessive focus on universal approximation properties could blur the truly important aspects of deep networks for image classification.


B. Weak points:

- There is a bit of a discontinuity in the paper when the scattering transform is introduced. There seem to be two almost independent messages: in the first part of the paper different contractive activations are evaluted, intuition is given, theoretical results are derived for soft thresholding. In the second part of the paper a complex scattering transform is introduced. Now it is not anymore clear how to define soft thresholding or a ReLU---only the modulus is obvious. The experiments are not anymore evaluating improvements in the Fisher ratio across layers but rather demonstrating that a particular network with fixed filters and learned channel combinations classifies well. It is not clear how the findings of the first part of the paper inform the second part (save for the fact that the used wavelet frame is tight and so are the 1x1 convolutions). I would appreciate if the narrative would make a better, more organic connection between the two parts.


C. Recommendation:

I recommend that the paper be accepted. It provides an original and convincing insight into the internal mechanics of deep networks for image classification.


D. Questions / suggestions:

- In the case of soft thresholding, the bias term is implicit in the nonlinearity. It is also learned from data by computing the standard deviation of <x, f_m>. Thus it does not seem completely true that this network does not have any learnable bias terms.
- From the last paragraph of 2.1 it seems that a similar implicit bias is also used for ReLU activations. Is this true? How is the threshold \lambda set in this case? If it is true, then it seems different from certain existing bias-free networks whose linearizations are indeed linear (as opposed to affine) operators, e.g. the cited Mohan et al. (2019).
- In my opinion the first paragraph in 2.1. could use some rewriting / refactoring. The streamlining might have went to far. E.g. one wants to identify only one out of C components, x is a mixture random vector, components are equiprobable, not x. It could be friendly to the reader to write down the mixture model (that is to say the density), ... Similar remarks hold throughout the manuscript.
- Perhaps I am misreading (1), but to me it seems that one way it can be made large is that all classes have the same distribution with a large \mu_c and small covariance. I suppose the key is that one optimizes over contractive maps such as projections, which don't allow the means to grow. Is this right? Is it a motivation to use Parseval frames?
- Beyond theory, especially in the numerics, it seems to me that one could do strictly better by relaxing the tightness constraint (provided sufficient training data). Would this further improve the numerics?
- At the end of Section 2 the authors state that characterizing the mean transformations of ReLU and modulus brings about considerable difficulties. Could we have a sense of those difficulties?
- In the third paragraph under "Choice of contraction", perhaps it is worth mentioning that not all filters f_m can be bandpass as otherwise F could not be a frame.
- In 3.1, I am confused by the sentence after the last display on page 6 and the significance of "o": "Each R_j is ... by cascading more than o modulus". Could you explain this better? (nb: plural is moduli). 


E. Minor comments

- full stop missing after display in Theorem 2.1
- bottom of p4: concentrations -> contractions?
- The assumption that E(\| x \|^2) = d under "Choice of contraction" should probably appear close to the first paragraph of 2.1 




review_title: Intriguing insights into class mean/variance evolution in deep networks

Meta-review: After reading the author’s response, all reviewers recommend accepting the paper. 

The authors provided an extensive response carefully considering all reviewers' comments. After incorporating the feedback, the manuscript improved in terms of presentation, relation to the literature and empirical results.

The paper is very well written and motivated. On top of the insightful analysis, experimental results are strong, obtaining comparable performance to that of a ResNet-18 on ImageNet.

R1 and R3 strongly support the paper while R2 and R4 consider it borderline.

R2 raised questions about experimental details and reproducibility. While R2 did not comment, these concerns were very clearly addressed by the authors in the view of the AC.

R4 was initially concerned with the novelty of the approach, but changed their mind after the author's response. The AC encourages the authors to further consider the feedback provided by the reviewer after the discussion period was over.

Decision: Accept (Poster)