Separation and Concentration in Deep Networks
Authors: John Zarka
Florentin Guth
Stéphane Mallat
Venue: ICLR 2021 Poster (2021)
Keywords: fisher ratio
neural collapse
mean separation
concentration
variance reduction
deep learning
image classification

Abstract: Numerical experiments demonstrate that deep neural network classifiers progressively separate class distributions around their mean, achieving linear separability on the training set, and increasing the Fisher discriminant ratio. We explain this mechanism with two types of operators. We prove that a rectifier without biases applied to sign-invariant tight frames can separate class means and increase Fisher ratios. On the opposite, a soft-thresholding on tight frames can reduce within-class variabilities while preserving class means. Variance reduction bounds are proved for Gaussian mixture models. For image classification, we show that separation of class means can be achieved with rectified wavelet tight frames that are not learned. It defines a scattering transform. Learning  $1 \times 1$ convolutional tight frames along scattering channels and applying a soft-thresholding reduces within-class variabilities. The resulting scattering network reaches the classification accuracy of ResNet-18 on CIFAR-10 and ImageNet, with fewer layers and no learned biases.

Published as a conference paper at ICLR 2021
SEPARATION AND CONCENTRATION IN DEEP NET- WORKS
John Zarka, Florentin Guth
Département d' d'informatique de I'ENS, ENS, CNRS, PSL University, Paris, France (john. zarka, florentin guth) @ens. fr
Stéphane Mallat
Collège de France, Paris, France Flatiron Institute. New York, USA
ABSTRACT
Numerical experiments demonstrate that deep neural network classifiers progres- sively separate class distributions around their mean, achieving linear separability on the training set, and increasing the Fisher discriminant ratio. We explain this mechanism with two types of operators. We prove that rectifier without biases applied sign-invariant tight frames can separate class means and increase Fisher ratios. On the opposite, a coftthressoldding on tight frames can reduce within- class variabilities while preserving class means. Variance reduction bounds are proved for Gaussian mixture models. For image classification, we show that sep- aration of class means can be achieved with rectified wavelet tight frames that are not learned. It defines a scattering transform. Learning x convolutional tight frames along scattering channels and applying a soft-thresholding reduces within-class variabilities. The resulting scattering network reaches the classifica- tion accuracy of ResNet-18 on CIFAR-10 10 and ImageNet, with fewer layers and no learned biases.
INTRODUCTION
Several numerical works (Oyallon 2017 Papyan 2020 Papyan et al., 2020) have shown that deep neural networks classifiers (LeCun et al. 2015) progressively concentrate each class around sep- arated means, until the last layer, where within-classes variability may nearly "collapse" (Papyan) al. 2020). The linear separability of a class mixture is characterized by the Fisher discriminant ratio Fisher 1936 Rao 1948). The Fisher discriminant ratio measures the separation class means relatively to the variability within each class, measured by their covariances. The neu- ral collapse appears through a considerable increase of the Fisher discriminant ratio during training (Papyan et al. 2020).. No mathematical mechanism has yet been provided explain this separation and concentration of probability measures.
Linear separability and Fisher ratios can be increased by separating class means without increasing the variability of each class, or by concentrating each class around its mean while preserving the mean separation. This paper shows that these separation or concentration properties can be achieved with one-layer network operators using different pointwise non non- non-linearities. We cascade these opera- tors define structured deep neural networks with high classification accuracies, and which can be analyzed mathematically.
Section 2 studies two-layer networks computed with a linear classifier applied pF, where F linear and is a pointwise non-linearity. First, we show that pF can separate class means with ReLU p.(u) = max(u, u, 0) and a sign-invariant F. We prove that P then increases the Fisher ratio. As in Parseval networks (Cisse et al. 2017), F' is normalized by imposing that it is tight frame which satisfies FT F Id. Second, to concentrate the variability each class around its mean, we shrinking non-linearit implemented by soft-thresholding Pt. For Gaussian mixture mod- els, we prove that F concentrates within-class variabilities while nearly preserving class means, under appropriate sparsity hypotheses. A linear classifier applied to these pF defines two-layer

---

Published as a conference paper at ICLR 2021
neural networks with learned bias parameters in the hidden layer, whose properties are studied mathematically and numerically.
Cascading several convolutional tight frames with ReLUs soft-thresholdings defines deep ral network which progressively separates class means and concentrates their variability, One may wonder if can avoid learning these frames by using prior information the geometry of im- ages. Section 3 shows that the class mean separation can be computed with wavelet tight frames, which
learned. They separate scales, directions and phases, which are known groups of transformations. A cascade of wavelet filters and rectifiers defines a scattering transform (Mallat
2012). which has previously been applied image classification (Bruna Mallat 2013] Oyal- lon & Mallat 2015). However, such networks do not reach state-of-the-an classification results. We show that important improvements are obtained by learning
convolutional projectors and tight frames, which concentrate within-class variabilities with soft-thresholdings. It defines bias-free deep scattering network whose classification accuracy reaches ResNet- 18 (He et al.)
2016) CIFAR-10 and ImageNet. Code to reproduce all experiments of the paper is available at com/ -ar-zzrkaaaa http:///eepppaaaioooooncnnttaaae
The main contributions of this paper are:
A double mathematical mechanism to separate and concentrate distinct probability mea- sures, with rectifier and soft-thresholding applied to tight frames. The increase of Fisher ratio is proved for tight-frame separation with rectifier. Bounds on within-class covariance reduction are proved for noftthreshholing on Gaussian mixture models.
The introduction of bias-free scattering network which reaches ResNet-18 accuracy CIFAR-10 and ImageNet. Learning reduced to 1 x convolutional tight frames which concentrate variabilities along scattering channels.
CLASSIFICATION BY SEPARATION AND CONCENTRATION
The last hidden layer of neural network defines representation (xx), to which is applied linear classifier. This section studies the separation of class means and class variability concentration for pf in a two-layer network.
2.1 TIGHT FRAME RECTIFICATION AND THRESHOLLING
We begin by briefly reviewing the properties of linear classifiers and Fisher discriminant ratios. We then analyze the separation and concentration of Φ pF, when p is a rectifier or a coft-thresholding and F is tight frame.
Linear classification and Fisher ratio We consider a random data vector x IRd whose class labels are v(x) E (1,...,C), C). Let xe be random vector representing the class c, whose probabil- ity distribution is the distribution of x conditioned by y(z) = c. We suppose that all classes are equiprobable for simplicity. Ave, denotes C-1 ΣE
We compute representation of x with an operator & which is standardized, that E(+(x)) and each coefficient of ((x) has a unit variance. The class means μ4. = E(((x.)) thus satisfy E.Ho = A linear classifier (W, b) on ((x) returns the index the maximum coordinate of ( E RC, An optimal linear classifier (W,)) minimizes the probability of a classification error. Optimal linear classifiers training data. Neural networks often use logistic linear classifiers, which minimize a cross-entropy loss. The standardization of the last layer (xx) is implemented with batch normalization (loffe & Szegedy 2015).
estimated by minimizing regularized loss function the
A linear classifier have small error if the typical sets of each ((x22) have little overlap, and in particular if the class means H4 = (xx)) are sufficiently separated relatively to the variability of each class. Under the Gaussian hypothesis, the variability of each class is measured by the covariance Σg of ((x.). Let Σw = Ave, Σx be the average within-class covariance and Σn = Ave, Me 2²~ be the between-class covariance of the means. The within-class covariance can be
whitened and normalized to Id by transforming (2:) with the square root E of En-. All classes
2

---

Published as a conference paper at ICLR 2021
c, c. are highly separated if || Me -Ew² per > 1. This separation is captured by the Fisher
discriminant ratio 2w B.. We shall measure its trace:
C-1 (r(2w/EB) Ave 2 Me
(1)
Fisher ratios have been used to train deep neural networks replacement for the cross-entropy loss (Dorfer et al. 20133 Stuhlsatz et 2012 Sun et al. 2019 Wu et al. 2017 Sultana et al., 2018
Li al. 20T6). In this paper, we use their analytic expression analyze the improvement of linear
classifiers.
Linear classification obviously cannot improved with linear representation 4. The following proposition gives simple condition to improve (or maintain) the error of linear classifiers with non-linear representation.
Proposition 2.1. If has linear inverse, then it decreases (or maintains) the error of the optimal linear classifier, and it increuses (or maintains) the Fisher ratio
To prove this result, observe that if &b has linear inverse & ! then Wr ('((x) with W W The minimum classification error by optimizing W is thus above the error obtained by optimizing W" Appendix proves that the Fisher ratio 1 is also increased or preserved.
There are qualitatively two types of non-linear operators that increase the Fisher ratio E''B.. Sep- aration operators typically increase the distance between the class means without increasing the vari- ance Σw within each class. We first study such operators having linear inverse, which guarantees through Proposition 2.1 that they increase the Fisher ratio. We then study concentration operators which reduce the variability Σw with non-lineat shrinking operators, which are not invertible. It will thus require a finer analysis of their properties.
Separation by tight frame rectification Let 1 = pF be an operator which computes the first layer of neural network, where p is pointwise non-linearity and F is linear. We first study sep- aration operators computed with a ReLU pr((u) = max(u, 0) applied to an invertible sign-invariant matrix. Such a matrix has rows that can be regrouped in pairs of opposite signs. It can thus be written F = [- FTTT where f is invertible. The operator pF separates coefficients according to their sign. Since (=(u)) Pr(-u) u, it results that + = pr F is linearly invertible. According to Proposition 2.1 it increases (or maintains) the Fisher ratio, and we want to choose F maximize this increase.
We can thus normalize the rows fm of F without affecting linear classification performance. To ensure that F E Rexd is invertible with stable inverse, we impose that it is normalized tight frame of IRd satisfying
Observe that (au) = ap, if 2
F F Id and |l 2 d/p for 1 S.
The tight frame can be interpreted as a rotation operator in higher dimensional space, which aligns the axes and the directions along which Pr performs the sign separation. This rotation must be adapted order to optimize the separation class means. The fact that F a tight frame can be interpreted normalization which simplifies the mathematical analysis.
Suppose that all classes z, of x have a Gaussian distribution with a zero mean μ!c = 0, but different covariances E.. These classes are not linearly separable because they have the same mean, and the Fisher ratio is 0. Applying F. can separate these classes and improve the Fisher ratio. Indeed, z is a zero-mean Gaussian random variable, then E(max 0)) = -2.T) / so we verify that for F [T FTTT.
B(p,Fr.) (2x) (diag(FE, 12:))11(((((a(((2227F) ""
The Fisher ratio can then be optimized by maximizing the covariance ΣB between the mean vector components diag( FE.FT) for all classes c. we know a priori that that Te and -I, have the same probability distribution, as in the Gaussian example, then we can replace p, by the absolute value Pa (u) |Iu| = Pr(u)) (-u., and pr F by pa F, which reduces by 2 the frame size.
3

---

Published as a conference paper at ICLR 2021
Concentration by tight frame soft-thresholding If the class means of x are already separated, then we can increase the Fisher ratio with a non-linear that concentrates each class around its mean. The operator & must reduce the within-class variance while preserving the class separation. This can be interpreted as a non-linear noise removal if we consider the within-class variability as additive noise relatively the class mean. It can be done with soft-threshodding estimators introduced in Donoho & Johnstone 1994). A soft-thresholding pr(u) = sign(u) max(|ul - X, 0) shrinks the amplitude of u by À in order reduce its variance, while introducing a bias that depends on A. Donoho & Johnstone (994)) proved that soft-thresholding estimators are highly effective to tight frame F.
estimate signals that have a sparse representation
To evaluate more easily the effect of a tight frame soft- noftthhresholding on the class means, we apply the linear reconstruction Ff on pe Fx, which thus defines a representation (xx) FT pe Fx. For a strictly positive threshold, this operator is not invertible, so we cannot apply Proposition 2. to prove that the Fisher ratio increases. We study directly the impact of on the mean and covariance of each class. Let Ze be the vector representing the class The mean Me [xx,) is transformed into j.. (x)()))) and the covariance Eg of Te into the covariance E. of ((t.). The average covariances are Σw Ave, E, and Σw = Ave, Σe.
Suppose that each ze is a Gaussian mixture, with a potentially large number of Gaussian components centered at Moc. with fixed covariance Id
Pe Σ Tork No.k-N Id)).
This model is quite general, since it amounts covering the typical set of realizations of z, with union of balls of radius a, centered in the (Ha,))s. The following theorem relates the reduction of within-class covariance the sparsity of F Fec,. It relies on the softtthresholding estimation results of Donoho & Johnstone (1994).
For simplicity, we suppose that the tight frame is an orthogonal basis, but the result can be extended to general normalized tight frames. The sparsity is expressed through the decay of sorted basis coefficients. For a vector = E Rd, we denote ((r) a coefficient of rank r: |=(-)] 2 |2(++1| for S r S d. The theorem imposes a condition on the amplitude decay of the (F(e,)))(t when r increases, which is sparsity measure. We write a(r) ~ b(r) if C1 a(r) S b(r) S C2 a(r) where Cj and C2 do not depend upon d nor a. The theorem derives upper bounds on the reduction of within-class covariances and on the displacements of class means. The constants do not depend upon d when it increases to co nor on when it decreases to O.
Theorem 2.2. Under the mixture model hypothesis 2 we have:
Tr(Ew) = Tr(Mm) + oz2d d, with Tr(EM) C-1
Σ
Te, l| - Mo
(3)
c,k
If there exists > 1/2 such that [(FMc,)) ~ -= then a tight frame soft-thresholding with threshold X = a /2logg satisfies:
( log d),
(4)
and all class means satisfy:
lIne 12 0 -((22// log d).
(5)
Under appropriate sparsity hypotheses, the theorem proves that applying &b F pTF' reduces considerably the trace of the within-class covariance. The Gaussian variance o²d d is dominant in 3
and is reduced to 2(02-1/a log d) in 4 The upper bound (5) also proves that FT p.F creates relatively small displacement of class means, which is proportional to log d. This is important to
ensure that all class means remain well separated. These bounds qualitatively explains the increase of Fisher ratios, but they are not sufficient to prove a precise bound on these ratios.
In numerical experiments, the threshold value of the theorem is automatically adjusted as follows. Non non-asymptotic optimal threshold values have been tabulated as a function of by Donoho & John- stone (1994). For the range of d used in our applications, a nearly optimal threshold is À 1.5 σ.. We rescale the frame variance o² by standardizing the input x so that it has a zero mean and each coefficient has unit variance. In high dimension d, the within-class variance typically dominates
4

---

Published as a conference paper at ICLR 2021
the variance between class means. Under the unit variance assumption have Tr(Ew) d. If F E RPxd ix is a normalized tight frame then we also verify as in (3) that Tr(Ew) v o o'p o² d/p.
It results that we choose À = 1.5 vd/p.
A soft-thresholding can also be computed from ReLU with threshold Pre max(u because Px (u) = Prr(u) P-r(-u). It results that -== [F==F"]pr Pre --= F² F. How- ever, a thresholded rectifier has more flexibility than theehhhresho,,ngg because may recombine differently Prt and Prc( to also separate class means, explained previously. The choice of threshold then becomes trade-off between separation of class means and concentration of class variability. numerical experiments, we choose lower À vd/p p for ReLU with a threshold.
2.2 TWO-LAYER NETWORKS WITHOUT BIAS
We study two-layer bias-free networks that implement linear classification on pF' where F is a normalized tight frame and p may be rectifier, an absolute value or a soft-thresholding, with no learned bias parameter. Bias-free networks have been introduced for denoising in Mohan et al. 2019), as opposed to classification or regression. We show that such bias-fre networks have lim- ited expressivity and do not satisfy universal approximation theorems (Pinkus 1999 Bach 2017). However, numerical results indicate that their separation and contractions capabilities are sufficient to reach similar classification results as two-layer networks with biases on standard image datasets.
Applying a linear classifier on (x)) computes:
( + b W WpFz +
This two-layer neural network has no learned bias parameters the hidden layer, and we impose that FFF Id with frame rows (fm)m having constant norms. As result, the following theorem proves that it does not satisfy the universal approximation theorem. We define a binary classifica- tion problem for which the probability of error remains above 1/4 for any number p neurons in the hidden layer. The proof is provided in Appendix for ReLU Pre with any threshold. The theorem remains valid with an absolute value Pa or a soft-thresholding (rs because they are linear combinations of Prt-
Theorem 2.3. Let X 2 0 be a fixed threshold and Pri(u) = max(u - O) Let F be the set of There exists random vector x Fd which
matrices F RRxd with bounded rows [l admits a probability density supported on the unit ball, and O function h: Red - TR. such that, for all p 2 d:
inf WERIKPFEF..EE
PPssgn(W (W Fx + b) + sgn(h(z)) 4
Optimization The parameters W, F and b are optimized with a stochastic gradient descent that 1d. following the optimiza- tion of Parseval networks (Cisse et al. 2017), after each gradient update of network parameters, insert second gradient step minimize a/2 F F Id]l This gradient update is:
minimizes logistic cross-entropy loss on the output. To impose F FTF=
F F+ FFFFFF
We also make sure after every Parseval step that each tight frame row Sm keeps a constant norm [l.mm || vd/p by applying a spherical projection: fm + dd/p fm/llfmlll These steps are per- formed across all experiments described in the paper, which ensures that all singular values of every learned tight frame are comprised between 0.99 and 1.01.
To reduce the number of parameters of the classification matrix W E RCxp, we factorize W W" F" with W" E CCdd It amounts to reprojecting pF in Fd with the oomi-orthogonal frame synthesis Ff and thus defines:
(τr) F Fa.
A batch normalization is introduced after & to stabilize the learning of W"
Image classification by separation and concentration Image classification is first evaluated on
the MNIST (LeCun al. 2010) and CIFAR-10 (Krizhevsky 2009) image datasets. Table T gives

---

Published as a conference paper at ICLR 2021
Table 1: For MNIST and CIFAR-10, the first gives the logistic classification and the second row the Fisher ratio Q for different signal representations (z). Results are evaluated with an absolute value Pa. a sotthhhresholding Pt. and a ReLU with threshold Prt.
F pF
(r(x)
Pa
Pt
p Prt
Error (%)
7.4
1.3
1.4
1.3
0.8
MNIST
Fisher
19
68
69
67
130
Error (%)
60.5
28.1
34.8
26.5
27.7
CIFAR
Fisher
6.7
15
13
16
12
the results of logistic classifiers applied the input signal x and to ((x)) = FT pF for different non-linearities p: absolute value Pa. soft offtthresholddnn Pt. and ReLU with threshold Pre. The tight frame F is a convolution on patches of size & x with stride of k/2, with k = 14 for MNIST and k for CIFAR. The tight frame F maps each patch to vector of larger dimension, specified in Appendix D Figure in Appendix shows examples learned tight frame filters.
On each dataset, applying F" pF greatly reduces linear classification error. which also appears with an increase of the Fisher ratio. For MNIST, all non-linearities produce nearly the same classi- fication accuracy, but CIFAR, the oft-thresholding has higher of MNIST are distinct averaged digits, which are well separated, because all digits are centered in the image. Concentrating variability with a soft-thresholding then sufficient. On the opposite, the classes of CIFAR images define nearly stationary random vectors because of arbitrary translations. As consequence, the class means Pe are nearly constant images, which are only discriminated by their average color. Separating these class means is then important for improving classification. As explained in Section 2. this is done by a ReLU Prs or this case an absolute value Pas which reduces the error. The ReLU with threshold Prt can interpolate between mean separation and vari- ability concentration, and thus performs usually at least as well as the other non-linearities.
Indeed, the class means
The error of the bias-free networks with a ReLU and absolute value are similar to the emrors obtained by training two-layer networks similar sizes but with bias parameters: 1.6% error on al. 2003), and 25% CIFAR-10 (Krizhevsky) 2010). It indicates that the elimination of bias parameters does not affect performances, despite the existence the counter- examples from Theorem that that cannot be well approximated by such architectures. This means that image classification problems have more structure that are not captured by these counter-examples, and that completeness in linear high-dimensional functional spaces may not be key mathematical properties to explain the preformances of neural networks. Figure | in Appendix shshows that the learned convolutional tight frames include oriented oscillatory filters, which also often the case of the first layer of deeper networks (Krizhevsky et al. 2012). They resemble wavelet frames, which are studied in the next section.
MNIST (Simard
3 DEEP LEARNING BY SCATTERING AND CONCENTRATING
To improve classification accuracy, cascade mean separation and variability concentration opera- tors, implemented by ReLUs and ooftthhresholdings on tight frames. This defines deep convolutional networks. However, show that some tight frames do not need to be learned. Section 3.1 reviews scattering trees, which perform mean separation by cascading ReLUs on wavelet tight frames. Sec- tion 3.2 shows that reach high classification accuracies by learning projectors and tight frame soft-thresholdingss, which concentrate within-class variabilities along scattering channels.
3.1 SCATTERING CASCADE OF WAVELET FRAME SEPARATIONS
Scattering transforms have been introduced to classify images by cascading predefined wavelet fil- ters with a modulus or a rectifier non-linearing (Bruna & Malaa 2013). We write as a product of wavelet tight frame rectifications, which progressively separate class means.

---

Published as a conference paper at ICLR 2021
Wavelet frame A wavelet frame separates image variations at different scales, directions and phases, with a cascade of filterings and subsamplings. We use steerable wavelets (Simoncelli & Freemann 1995) computed with Morlet filters (Bruna & Mallat 2013). There is one low-pass filter 9o. and I. complex band-pass filters having an angular direction é = L for 0 < l S L. These
filters can be adjusted (Selesnick al. 2005) so that the filtering and subsampling:
Fur(nn () ge(2n)
defines complex tight frame Fu- Fast multiscale wavelet transforms are computed by cascading the filter bank Fw on the output of the low-pass filter go (Mallat 2008
Each complex filter ge is analytic, and thus has a real part and imaginary part whose phases are shifted by = 7/2. This property is important to preserve equívariance to translation despite the subsampling with a stride of 2 (Selesnick et al. 2005). To define a sign-invariant frame as in Section 2.1] we must incorporate filters of opposite signs, which amounts to shifting their phase by We thus associate F real sign-invariant tight frame Fw by considering separately the four phases a = 0, 1/2, n. / It is defined by
Fw x (t,aa
with gr.0 = 2 2-222 /2 = 2-1/2Tmmg((e and De,t -tc. We apply rectifier Pr to the output of all real band-pass filters Je but not to the low-pass filter:
(x (o())) Pr * (t.))2))))
p, w
n,a.
The use of wavelet phase parameters with rectifiers is studied Mallat et al. 2019). The operator Pr Fu is linearly invertible because Fw is tight frame and the ReLU is applied to band-pass pass filters, which come in pairs of opposite sign. Since there are 4 phases and subsampling with a stride of 2, F is (L + 1/4) times larger than x.
Scattering tree A full scattering tree Sr of depth J/ is computed by iterating J times over P. Since each Fw has linear inverse, Proposition 2.1 proves that this separation can only increase the Fisher ratio. However it also increases the signal size by (I. + 1/4)", which is typically much too large. This is avoided with orthogonal projectors, which perform a dimension reduction after applying each F.
A pruned scattering tree S7 of depth J and order is defined in Bruna & Mallat (2013) as a convo- lutional tree which cascades J rectified wavelet filter banks, and at each depth prunes the branches with P, to prevent exponential growth:
Sr PI, f=1
After the ReLU, the pruning operator Pj eliminates the branches of the scattering which cascade more than o band-pass filters and rectifiers, where is the scattering order (Bruna & Mallat 2013). After J cascades, the remaining channels have thus been filtered by at least f-o successive low-pass filters 9o- We shall use a scattering transform of order o = 2. The operator P, also averages the rec- tified output of the filters Gt. along the phase a, for E fixed. This averaging eliminates the phase. It approximatively computes a complex modulus and produces a localized translation invariance. The resulting pruning and phase average operator P3 is x convolutional operator, which reduces the dimension of scattering channels with an orthogonal projection. If x has d pixels, then Sr(x)(m, k] 2 an array of images having 2-2 d pixels each channel k. because of the J subsamplings with stride of 2. The total number of channels K is 1 + JL + J[J 1)L2 /2. Numerical experiments are performed with wavelet filters which approximate Gabor wavelets (Bruna & Mallat 2013), with
== 8 directions. The number of scales J depends upon the image size. it J = 3 for MNIST and CIFAR, and J = 4 for ImageNet, resulting in respectively K = 217, 651 and 1251 channels.
Each Fw can only improve the Fisher ratio and the linear classification accuracy, but it is not guaranteed that this remains valid after applying P.. Table I gives the classification error of a logistic classifier applied on Sz)), after 1 x 1 orthogonal projection to reduce the number of channels, and a spatial normalization. This error is almost twice smaller than a two-layer neural network MNIST, given in Table II but it does not improve the error on CIFAR. On CIFAR, the error obtained by a ResNet-22 20 is times lower than the one of classifier on S (z). The main issue is now to understand where this inefficiency comes from.
7

---

Published as a conference paper at ICLR 2021
Table 2: Linear classification error and Fisher ratios ① of several scattering representations, on CIFAR-10 and ImageNet. For Sc, results are evaluated with a soft-thresholding Pr and thresholded rectifier Pre. The last column gives the error of ResNet-20 for CIFAR-I0 (He et al.) 2016) and 18 for ImageNet, taken from Lorchvision/mooee html
ort. docs stable/
ST
Sp
Sc (pt)
Sc (prt)
ResNet
Error (%)
27.7
12.8
8.0
7.6
8.8
CIFAR
Fisher
12
20
43
41
Top-5
54.1
20.5
11.6
10.7
10.9
Error (%)
ImageNet
Top-1
73.0
42.3
31.4
29.7
30.2
Fisher
2.0
18
51
44
3.2 SEPARATION AND CONCENTRATION IN LEARNED SCATTERING NETWORKS
A scattering iteratively separates class means with wavelet filters. Its dimension is reduced by predefined projection operators, which may decrease the Fisher ratio and linear separability. To avoid this source of inefficiency, we define scattering network which learns these projections. The second step introduces tight frame thresholdings along scattering channels, to concentrate within- class variabilities. Image classification results are evaluated on the CIFAR-10 (Krizhevsky 2009) and ImageNet (Russakovsky et al. 2015) datasets.
Learned seattering projections Beyond scattering trees, the projections P3 of a scattering trans- form 7 can be redefined as arbitrary orthogonal 1 x convolutional operators, which reduce the number of scattering channels: P,P Id. Orthogonal projectors acting along the direction index l of wavelet filters can improve classification (Oyallon & Mallat 2015). We these linear operators together with the final linear classifier. Before computing this projection, the mean and variances of each scattering channel is standardized with batch normalization Bv, by and 3 = O. This projected scattering operator can be written:
going to learn
setting affine coefficients
Sp = II BN pr Fw. j=1
Applying a linear classifier to Sp(x) defines a deep convolutional network whose parameters are the 1 x convolutional Pj and the classifier weights W, b. The wavelet convolution filters in F. not learned. The orthogonality of P, is imposed through the gradient steps (6) applied to F pf Table 2 that learning the projectors P, more than halves the scattering classification error of Sp relatively to Sx CIFAR- 10 and ImageNet, reaching AlexNet accuracy on ImageNet, while achieving higher Fisher ratio.
The learned orthogonal projections P3 create invariants families of linear transformations along scattering channels that depend upon scales, directions and phases. They correspond to image trans- formations which have been linearized by the scattering transform. Small diffcomorphisms which deform the image are examples of operators which are linearized by a scattering transform (Mal- lat 2012 The learned projector eliminates within-class variabilities which are not discriminative across classes. Since it is linear, it does not improve linear separability or the Fisher ratio. It takes advantage of the non- non-linear separation produced by the previous scattering layers.
The operator P3 is a projection on family of orthogonal directions which define new scattering channels, and is followed by a wavelet convolution F., along spatial variables. defines separable convolutional filters F.. along space and channels. Learning P, amounts to choosing orthogonal directions so that Fr, optimizes the class means separation. If the class distributions are invari- ant by rotations, the separation can be achieved with wavelet convolutions along the direction index OOyallon& & Mallat 2015),, but better results are obtained by learning these filters. This separable scattering architecture is different from separable approximations of deep network filters in discrete cosine bases (Ulicny et al., 2019) or in Fourier-Bessel bases (Qiu et al. 2018). A wavelet scattering computes p, F.. opposed to separable decomposition p, Fi.. the ReL.U is applied in
8

---

Published as a conference paper at ICLR 2021
Table 3: Evolution of Fisher ratio across layers for the scattering concentration network Se with ReLU with threshold Pre. on the CIFAR dataset.
Layer 0
5
6
7
8
CIFAR
Fisher
1.8
11
13
11
15
15
22
25
40
higher dimensional space indexed by wavelet variables produced by F.. It provides explicit coordi- nates to analyze the mathematical properties, but also increase the number of learned parameters as shown in Table Appendix D
Concentration along scattering channels projected scattering transform can separate class means, but does not concentrate class variabilities. To further reduce classification errors, following Section 2.1 concentration is computed with a tight frame soft-thresholding F pe Fj. applied on
scattering channels. It increases the dimension of scattering channels with x 1 convolutional tight frame F,, applies a soft-thresholding Pt. and reduces the number of channels with the x convolutional operator Ff. The resulting concentrated scattering operator
Sc = P) (P, F.F.).
(8)
It has 2. layers, with odd layers computed by separating means with a ReLu Pr and even layers computed by concentrating class variabilities with a nof-threshooldig Pt- According Section 2.1 the soft-threshold
= 1.5/d/p. This tefthhhresholdig may be replaced by thresholded rectifier (u) = max(u - X, 0) with lower threshold x = vd/p. A logistic classifier is applied (cr)) The resulting deep network does not include any learned bias parameter, except in the final linear classification layer. Learning is reduced to the 1 x 1 convolutional operators P3 and F} along scattering channels, and the linear classification parameters.
Table 2 gives the classification errors of this concentrated scattering on CIFAR for J (8 layers) (12 layers). The layer dimensions are specified in Appendix D The number
and ImageNet for of parameters of the scattering networks are given in Table Appendix D This concentration step reduces the error Sc by about 40% relatively to a projected scattering 3p. ReLU thresholding Prt produces an error slightly below soft-threshoding both CIFAR-10 and ImageNet, and this error also below the errors of ResNet-20 for CIFAR and ResNet-18 for ImageNet. These errors are also nearly half the classification errors previously obtained by cascading scattering tree Sy. with several 1 X convolutional layers and large MLP classifiers (Zarka et al.] 2020 Oyallon et al. 2017). It shows that the separation and concentration learning must be done each scale rather than at the largest scale output. Table 3 shows the progressive improvement of the Fisher ratio measured at each layer of Sc on CIFAR-10. The transition from odd layer 2 1 to an even layer 2j results from F Fr. which always improve the Fisher ratio by concentrating class variabilities. The transition from 2j to 2I + done by Ps+1p,Fuw which may decrease the Fisher ratio because of the projection PI+1, but globally brings an important improvement.
CONCLUSION
We proved that separation and concentration of probability measures can be achieved with rectifiers and thresholdings applied to appropriate tight frames We also showed that the separation of class means can be achieved by cascading wavelet frames that are learned. It defines a scattering transform. By concentrating variabilities with a thresholding along scattering channels, we reach ResNet- classification accuracy CIFAR-10 10 and ImageNet.
major mathematical issue is to understand the mathematical properties of the learned projectors and tight frames along scattering channels. This is necessary understand the types of classification problems that are well approximated with such architectures, and to prove lower bounds on the evolution of Fisher ratios across layers.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This work was supported by grants from Région Il-nce-Faacc and the PRAIRIE 3IA Institute of the French ANR-19-P3IA-000) program. We would like to thank the Scientific Computing Core at the Flatiron Institute for the use of their computing resourees.
REFERENCES
M. Andreux, T. Angles, G. Exarchakis, Leonarduzzi, Rochette, Thiry, Zarka, S. Mallat, J. Andén, E. Belilovsky, J. Bruna, V. Lostanlen, M. J. Hirm, E. Oyallon, S. Zhang, C. E. Cella, and M. Eickenberg. Kymatio: Scattering transforms python. Journal of Machine Learning Research, 21(60):1-6, 2020.
F. Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research,
2017.
J. Bruna and S. Mallat. Invariant scattering convolution networks. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1872-1886, 2013.
M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. In Proceedings of the 34th International Conference on Ma- chine Learning- Volume 70, pp. 853, 2017.
D. L. Donoho and I. M. Johnstone. Ideal spatial adaptation by wavelet shrinkage. Biometrika, (3):425-455, 09 1994.
M. Dorfer, R. Kelz, and G. Widmer. Deep linear discriminant analysis. arXiv: 1511::551 2015.
arXiv preprint
R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(7); 179-188, 1936.
K. He, Zhang. S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on International Conference Machine Learning Volume 37, pp. 448-456, 2015.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
A. Krizhevsky. Convolutional deep belief networks on cifar-10, 2010.
A. Krizhevsky, Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012.
Y. LeCun, C. Cortes, and C.J. Burges, Mnist handwritten digit database. ATT Labs [Online]. Available: http://aamm.eeccn.comm/eeaee
2, 2010.
Y. LeCun, Y. Bengio, and G.E. Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
Y. Li, W. Zhao, and Pan. Deformable patterned fabric defect detection with fisher criterion-based deep learning. IEEE Transactions on Automation Science and Engineering, 14(2):1566-126, 2016.
Mallat. Wavelet Tour of Signal Processing. Third Edition: The Sparse Way. Academic Press, 3rd edition, 2008.
S. Mallat. Group invariant scattering. Communications Pure and Applied Mathematics, 65(10): 1331-1398, 2012.
Mallat, Zhang, and G. Rochette. Phase harmonic correlations and convolutional neural net- works. Information and Inference: A Journal of the IMA, 11 2019. doi: 10. 10.1093/imaiai/aaz0)
10

---

Published as a conference paper at ICLR 2021
S. Mohan, Z. Kadkhodaie, E. P. Simoncelli, and C. Srmnandez-Grandie Robust and inter- pretable blind image denoising via bias-frec convolutional neural networks. arXiv preprint arXiv:1906.05478, 2019.
E. Oyallon. Building a regular decision boundary with deep networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1886-1894, 2017.
E. Oyallon and S. Mallat. Deep roto- tot-ttranllttion scattering for object classification. In IEEE Con- ference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 2865-2873. IEEE Computer Society, 2015.
E. Oyallon, E. Belilovsky, and S. Zagonuyko. Scaling the scattering transform: Deep hybrid net- works. Proceedings of the IEEE International Conference on Computer Vision, pp. 5618 6627-567, 2017.
V. Papyan. Traces of class/cross-class structure pervade deep learning spectra, 2020.
V. Papyan, X. Y. Han, and D. L. Donoho. Prevalence neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 2020.
A. Pinkus. Approximation theory of the mlp model in neural networks. Acta Numerica, 8:143-195, 1999.
Q Qiu, X. Cheng, R. Calderbank, and G. Sapiro. DCFNet: Deep neural network with decomposed convolutional filters. International Conference on Machine Learning, 2018.
C. Rao. The utilization of multiple measurements in problems of biological classification. Journal 10(2):159-19) (
of the Royal Statistical Society: Series B (Methodological),
1948.
O. Russakovsky, J. Deng, H. Su, J. Krause. S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC)), 15(3):211 1-252, 2015.
I. W. Selesnick, R. G. Baraniuk, and N. C. Kingsbury. The dual-tree complex wavelet transform. IEEE signal processing magazine, 22(6)::23-151, 2005.
P. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to visual document analysis. Proceedings of the Seventh International Conference on Document Analysis and Recognition Volume 2, 2003.
E. P. Simoncelli and W. T Freeman. The steerable pyramid: A flexible architecture for multi- scale derivative computation. In Proceedings., International Conference on Image Processing, volume 3. pp. 444 447. IEEE, 1995.
A. Stuhlsatz, Lippel, and T. Zielke, Feature extraction with deep neural networks by a generalized discriminant analysis. IEEE Transactions on Neural Networks and Learning Systems, 23(4:5596 608, 2012.
N. Sultana, B. Mandal, and N. Puhan. Deep residual network with regularised fisher framework for detection of melanoma. IET Computer Vision, 1296);106611104, 2018.
K. Sun, J. Zhang, H. Yong, and J. Liu. Fpeanet: Fisher discrimination for principal component analysis network. Knowledge- Based Systems, 066:0081111 2019.
M. Ulieny. V. Krylov, and R. Dahyot. Harmonic networks for image classification. In Proceedings of the British Machine Vision Conference, Sep. 2019.
L. Wu, C. Shen, and Van Den Hengel. Deep linear discriminant analysis on fisher networks: A hybrid architecture for person re-identification. Pattern Recognition, 65:238-2555 252288220, 2017.
J. Zarka, L. Thiry, T. Angles, and S. Mallat. Deep network classification by scattering and homotopy dictionary learning. In International Conference on Learning Representations, 2020.
11


Reviews:
[{"review_id": "uY8kPP2gTA1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces structured deep network architectures\nthat can be analyzed mathematically and have high classification accuracies on\ncomplex image databases. The proposed mechanism consists of iterating over tight frame contractions. \nThey also show that spatial filters do not need to be learned, and can be defined from wavelet frames.\n\n############\n\nOverall inclined for accepting the paper although I am a bit hesitant due to the lack of experimental details and/or code. Overall, the \n main idea is interesting and novel and the paper is well written. \n\n############\n\nPros\n\n* Interesting idea and theoretical results/implications. \n\n############\n\nCons \n* Details on the experiments are largely missing, i.e. parameters etc are not listed anywhere in the experimental section.\n* No code provided or even mentioned in the paper. This makes the experimental verification harder. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "qCTALg_G2VC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting paper which seeks to explain the recently observed phenomenon in deep net image classification, whereby variance of activations for a fixed class becomes small / collapses in deeper layers, while the class means remain well-separated. The authors make an intriguing connection with the classical results of Donoho & Johnstone (1994) on soft thresholding to show that for a certain mixture of Gaussian mixtures, soft thresholding reduces the order of intra-class variation from O(d) to O(log d) while only moving the means by O(log d), provided that the means of the mixture components are compressible (approximately sparse) in the given frame.\n\n\nA. Strong points\n\n- The simplified framework proposed by the authors seems to indeed distill the key aspects of the problem and the results on soft thresholding are very promising. It is reassuring to see a connection to sparsity / compressibility which plays important but still elusive role in deep network classification.\n- Theorem 2.2. which uses the soft thresholding non-linearity gives an intriguing insight into how the above reduction of variance could occur. \n- The numerical experiments convincingly illustrate the theoretical discussions.\n- As a meta point, I very much agree with the statement that excessive focus on universal approximation properties could blur the truly important aspects of deep networks for image classification.\n\n\nB. Weak points:\n\n- There is a bit of a discontinuity in the paper when the scattering transform is introduced. There seem to be two almost independent messages: in the first part of the paper different contractive activations are evaluted, intuition is given, theoretical results are derived for soft thresholding. In the second part of the paper a complex scattering transform is introduced. Now it is not anymore clear how to define soft thresholding or a ReLU---only the modulus is obvious. The experiments are not anymore evaluating improvements in the Fisher ratio across layers but rather demonstrating that a particular network with fixed filters and learned channel combinations classifies well. It is not clear how the findings of the first part of the paper inform the second part (save for the fact that the used wavelet frame is tight and so are the 1x1 convolutions). I would appreciate if the narrative would make a better, more organic connection between the two parts.\n\n\nC. Recommendation:\n\nI recommend that the paper be accepted. It provides an original and convincing insight into the internal mechanics of deep networks for image classification.\n\n\nD. Questions / suggestions:\n\n- In the case of soft thresholding, the bias term is implicit in the nonlinearity. It is also learned from data by computing the standard deviation of <x, f_m>. Thus it does not seem completely true that this network does not have any learnable bias terms.\n- From the last paragraph of 2.1 it seems that a similar implicit bias is also used for ReLU activations. Is this true? How is the threshold \\lambda set in this case? If it is true, then it seems different from certain existing bias-free networks whose linearizations are indeed linear (as opposed to affine) operators, e.g. the cited Mohan et al. (2019).\n- In my opinion the first paragraph in 2.1. could use some rewriting / refactoring. The streamlining might have went to far. E.g. one wants to identify only one out of C components, x is a mixture random vector, components are equiprobable, not x. It could be friendly to the reader to write down the mixture model (that is to say the density), ... Similar remarks hold throughout the manuscript.\n- Perhaps I am misreading (1), but to me it seems that one way it can be made large is that all classes have the same distribution with a large \\mu_c and small covariance. I suppose the key is that one optimizes over contractive maps such as projections, which don't allow the means to grow. Is this right? Is it a motivation to use Parseval frames?\n- Beyond theory, especially in the numerics, it seems to me that one could do strictly better by relaxing the tightness constraint (provided sufficient training data). Would this further improve the numerics?\n- At the end of Section 2 the authors state that characterizing the mean transformations of ReLU and modulus brings about considerable difficulties. Could we have a sense of those difficulties?\n- In the third paragraph under \"Choice of contraction\", perhaps it is worth mentioning that not all filters f_m can be bandpass as otherwise F could not be a frame.\n- In 3.1, I am confused by the sentence after the last display on page 6 and the significance of \"o\": \"Each R_j is ... by cascading more than o modulus\". Could you explain this better? (nb: plural is moduli). \n\n\nE. Minor comments\n\n- full stop missing after display in Theorem 2.1\n- bottom of p4: concentrations -> contractions?\n- The assumption that E(\\| x \\|^2) = d under \"Choice of contraction\" should probably appear close to the first paragraph of 2.1 \n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fX2exoNJPZY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes using networks that are composed of tight frames to analyze the clustering property of networks across layers.\nYet, the main focus of the paper in the first part is to construct the tight frame-based networks and then in the second part to train scattering transforms based networks. \nWhile the ideas are interesting I have several concerns:\n\n1. The idea of encouraging tight frame structure is not new and appeared already in several works in the literature.\n2. The idea of training a scattering transform is not new and has been done before. For example, there is a work by Mallat that shows that one may just add 1x1 convs to the scattering transform and train it. So the current work is not of much difference. \n\nGiven these two concerns, I don't think the current novelty is sufficient for publication. \n\n=========================================================================================\n\nUpdated review:\nThank you for the clarification. The previous version was indeed confusing to me. I have raised my score although I think some points still need to be addressed in the revision following my previous comments as they were not fully addressed nor in the response neither in the revision: \n1.  The concern with respect to previous works is not only regarding Parseval networks. There are other more recent works that use orthogonality constraints on the network. Such examples include\nhttps://ieeexplore.ieee.org/document/8877742\nhttps://openaccess.thecvf.com/content_cvpr_2017/papers/Xie_All_You_Need_CVPR_2017_paper.pdf\nhttps://proceedings.neurips.cc/paper/2018/hash/bf424cb7b0dea050a42b9739eb261a3a-Abstract.html\n\nAll these works show a similar observation to the one claimed in the paper that by using orthogonality (or frame-like) operators one may train a network without skip connections and get similar results. \nIndeed, in the paper, more observations are being made that are different than what is presented in these works but a more proper comparison should be made. \n\n2. This is the work the authors should look at by Mallat\nhttps://arxiv.org/pdf/1809.06367.pdf\nThey get similar performance to ResNet with a scattering transform-based network. \nIndeed, also here it is not exactly the same network that the authors here are using but there are remarkable similarities and these should be well addressed. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DtvMSsbVNYJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces the concatenation of a tight frame with a scalar non-expansive operator (mainly the modulus and soft-thresholding) as a unit, which - when applied to scattering transform coefficients - can yield high quality image classification results, improve the Fisher discriminant ratio, and (for soft-thresholding) allow some mathematical analysis in the sense of variance reduction bounds.\n\nThe paper is well-written, well-motivated, interesting and, to the best of my knowledge, novel. It combines theoretical insights with practical results that are only slightly worse than ResNet-18 on ImageNet. \n\nOn the negative side I only have a couple of minor things:\n- It is slightly confusing to me that the operator $\\rho$ is called \"contracting\" instead of \"non-expansive\". In the optimization literature a contraction would have to satisfy the stated property with an additional factor \\gamma < 1 on the right hand side. I think it is worth rephrasing this everywhere (even in the title) to avoid confusion. \n- I think it is worth stating how well the Parseval regularization enforces the orthogonality constraint for all numerical experiments. Why does $\\alpha$ vary quite strongly from experiment to experiment?\n- Is the batch normalization for stabilizing the learning of W' really necessary despite the rather well-behaved remaining architecture?\n- On the practical side, it would have been interesting to see if the remaining gap to ResNet can be closed if the fixed spatial wavelet filters are (partially) replaced by learnable ones. To highlight the advantages of avoiding to learn such filters, the overall number of learnable parameters or the training times could be compared. \n\nIn summary, I think this is an interesting paper that does merit a publication. \n\n\n----------------------\nAfter the rebuttal: I'd like to thank the authors for their answers, particularly for resolving the confusion about the term \"contraction\". I believe this is a good paper and stick to my rating of recommending its acceptance. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading the author’s response, all reviewers recommend accepting the paper. 

The authors provided an extensive response carefully considering all reviewers' comments. After incorporating the feedback, the manuscript improved in terms of presentation, relation to the literature and empirical results.

The paper is very well written and motivated. On top of the insightful analysis, experimental results are strong, obtaining comparable performance to that of a ResNet-18 on ImageNet.

R1 and R3 strongly support the paper while R2 and R4 consider it borderline.

R2 raised questions about experimental details and reproducibility. While R2 did not comment, these concerns were very clearly addressed by the authors in the view of the AC.

R4 was initially concerned with the novelty of the approach, but changed their mind after the author's response. The AC encourages the authors to further consider the feedback provided by the reviewer after the discussion period was over.

Decision: Accept (Poster)