Dual-Tree Wavelet Packet CNNs for Image Classification
Authors: Hubert Leterme
Kévin Polisano
Valérie Perrier
Karteek Alahari
Venue:  (2021)
Keywords: convolutional neural networks
wavelet packet transform
dual-tree wavelet packet transform
image classification
deep learning
image processing

Abstract: In this paper, we target an important issue of deep convolutional neural networks (CNNs) — the lack of a mathematical understanding of their properties. We present an explicit formalism that is motivated by the similarities between trained CNN kernels and oriented Gabor filters for addressing this problem. The core idea is to constrain the behavior of convolutional layers by splitting them into a succession of wavelet packet decompositions, which are modulated by freely-trained mixture weights. We evaluate our approach with three variants of wavelet decompositions with the AlexNet architecture for image classification as an example. The first variant relies on the separable wavelet packet transform while the other two implement the 2D dual-tree real and complex wavelet packet transforms, taking advantage of their feature extraction properties such as directional selectivity and shift invariance. Our experiments show that we achieve the accuracy rate of standard AlexNet, but with a significantly lower number of parameters, and an interpretation of the network that is grounded in mathematical theory.

Under review as a conference paper at ICLR 2021
DUAL-TREE WAVELET PACKET CNNs FOR IMAGE CLASSIFICATION
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we target an important issue of deep convolutional neural networks (CNNs) - the lack a mathematical understanding their properties. We present explicit formalism that is motivated by the similarities between trained CNN kernels and oriented Gabor filters for addressing this problem. The core idea is to constrain the behavior of convolutional layers by splitting them into succes- sion of wavelet packet decompositions, which are modulated by freely-trained mixture weights. We evaluate our approach with three variants of wavelet decom- positions with the AlexNet architecture for image classification as an example. The first variant relies on the separable wavelet packet transform while the other two implement the 2D dual-tree real and complex wavelet packet transforms, tak- ing advantage of their feature extraction properties such as directional selectivity and shift invariance. Our experiments show that we achieve the accuracy rate of standard AlexNet, but with significantly lower number of parameters, and an interpretation of the network that is grounded in mathematical theory.
INTRODUCTION
Deep convolutional neural networks (CNNs) have dramatically improved state-of-the-arr perfor- mances many domains such speech recognition, visual object recognition object detection (LeCun et al. 2015).. However, they are very resource-intensive and full mathematical understand- ing of their properties remains a challenging issue.
On the other hand, in the field of signal processing, wavelet and multi-resolution analysis are built upon a well-established mathematical framework. They have proven to be very efficient tasks such as signal compression and denoising (Mallat 2009). Moreover, wavelet filters have been widely
used as feature extractors for signal, image and texture classification (Laine & Fan 1993
Pittner &
Kamarthi 1999
Yen 2000 Huang & Aviyente 2008).
While both fields rely on filters to achieve their goals, the two approaches are radically different. In wavelet analysis, filters are specifically designed to meet very restrictive conditions, whereas CNNs use freely trained filters, without any prior assumption on their behavior. Nevertheless, in many computer vision tasks, CNNs tend to learn parameters that pretty similar to oriented Gabor filters in the first layer (Boureau et al., 2010 Yosinski et al. 2014). This phenomenon suggests that early layers extract general features such edges or basic shapes, which are independent from the task at hand.
Proposed approach In order to improve our understanding CNNs, we propose to constrain their behavior by replacing freely-trained filters by series of discrete wavelet packet decompositions modulated by mixture weights. We therefore introduce prior assumptions to guide learning and reduce the number of trainable parameters in convolution layers, while retaining predictive power.
The main goal of our work is to describe and interpret the observed behavior of CNNs with a sparse model, taking advantage of the feature extraction properties of wavelet packet transforms. By increasing control over the network, we pave the way for future applications in which theoretical guarantees are critical.

---

Under review as a conference paper at ICLR 2021
In this paper we describe our wavelet packet CNN architectures with mathematical formulation and introduce an algorithm to visualize the resulting filters. As proof of concept, we based our experiments AlexNet (Krizhevsky et al. 2012). Our choice was driven by the large kernels in its first layer and convolution operations performed with a downsampling factor of 4. This allows to perform two levels of wavelet decomposition without any additional transformation, and facilitates visual comparison with our own custom filters. Note however that most CNNs trained on natural image datasets exhibit the same oscillating patterns. We therefore believe that our work could be extended to other architectures with a few adaptations.
Related work In a similar spirit, a few attempts to combine the two research fields have been made in recent years. Wavelet scattering networks (Bruna & Mallat 2013) compute CNN-like cascading wavelet convolutions to get translation-invariant image representations that are stable to deforma- tion and preserve high-frequency information. They were later adapted to the discrete case using complex oriented wavelet frames (Singh & Kingsbury 2017). While these networks are designed from scratch and are totally deterministic, other approaches enhance existing networks with wavelet filter preprocessing embedding. The goal is either improve classification performance without
increasing the network complexity (Chang & Morgan 2014 Williams & Li 2016, [Fujieda et al.) 2017] Williams & Lil 20188] Lu et al.] 2018] Luan et al.] 2018), or replace freely-trained layers by more constrained structures implementing spectral filtering. Such models include Gabor filters in parallel to regular trainable weight kernels (Sarwar et al. 2017), wavelet scattering coefficients as the input of a CNN (Oyallon et al., 2018), or linear combinations of discrete cosine transforms
(Ulicny et al., 2019).
Our approach falls into this second category, although our design is based upon different CNN architecture, i.e., AlexNet. To our knowledge we are the first to introduce the dual-tree wavelet packet transform (DT-CWPT) (Bayram & Selesnick (008) in such context. Like the filters used in the above papers, wavelet packet transforms are well-localized in the frequency domain and share a subsampling factor over the output feature maps. A major advantage with our approach is sparsity: single vector (called conjugate mirror filter, or CMF) is sufficient to characterize the whole process. Moreover, like Gabor filters, DT-CWPT extracts oriented and shift-invariant features, but achieves this goal with minimal redundancy, while providing an efficient decomposition algorithm based on separable filter banks. Regarding the discrete cosine transform, its complexity is similar to DT- CWPT but lacks orientation properties. Our models therefore provide a sparser description of the observed behavior of convolutional layers. This is a step toward a more complete description of CNNs by using a small number of arbitrary parameters.
BACKGROUND
Notations In this paper, d-dimensional tensors are written with straight bold capital letters: Z E XAA where A; denotes the size of Z along its i-th dimension; the shape of Z is denoted
Ad) 2D matrices are written in italic: U E RAXB and ID vectors in bold lower-case letters: E RA, For the sake of legibility, indices are written between square brackets.
(Z) = (Ai
The convolution between two matrices U E RAXB and V E A'''''' is defined, for all m E AO.. 22 2] and n E [ B' - 2}. by (U * V) [m, n] = En, U [m - i, n - j] Vi, j]. Since some indices are negative or bigger than the matrix size, U and V must be ex- tended beyond their limits, either by setting all outside values to zero, by using a periodic or symmetric pattern. Practical implications of this choice will not be discussed in this paper.
For any U E A U denotes the flipped matrix: U [m, n] = U [A - (m + 1), B - (n + 1)]. The upsampling and downsampling operators are respectively denoted and L. For any a N
(U t a) [,, n = U a [m a) [m, n) = U [am, anl.. Finally, for any scalar z E R, we denote = + U = =J + U, where J R BAXB denotes the matrix of ones.
x if both m and n are divisible by a (= 0 otherwise), and (U +
Discrete wavelet packet transform (WPT) This is brief overview on the WPT algorithm Mal-- lat 2009). written as a sequence of matrix convolutions. An illustration of the transform is given in Appendix A.7
2

---

Under review as a conference paper at ICLR 2021
We will implicitly build discrete orthogonal basis in which any matrix X E RNXN can be de- composed. The basis is made of oriented 2D waveforms with high frequency resolution, which is an interesting property for feature extraction. Considering pair of conjugate mirror filters (CMFs) and g E R", build separable 2D filter bank, made of one low-pass filter G h h and three high-pass filters GO h g .G(2) =g. h and G = 9
We start the decomposition with D ( = X. Let us assume that for given j E N, the feature maps of wavelet packet coefficients at scale j. denoted D have already been computed for any E (0 - -13. Then we compute the wavelet packet coefficients at the coarser scale j + by decomposing each feature map D (A) into four smaller submatrices:
E ((...). D D (D.x) G( 22.
{D]m)} is representation of X from which the original
At each scale j > 0, the set of 4 matrices signal can be reconstructed. Figure T Tillustrates the WPT resulting filters with j 2.
Dual-tree complex wavelet packet transform (DT-CWPT) WPT has interesting properties such as sparse signal representation and vertical horizontal feature discrimination. However, it suffers from a lack of shift invariance and a poor directional selectivity. To overcome this, Kinggsbuy (200T) designed discrete wavelet transform in which input images are efficiently decomposed in a tight frame of complex oriented waveforms with limited redundancy. was generalized to the wavelet packet framework by |Bayram& & Selesnick 2208).
In a nutshell, let us assume that we have decomposed an input matrix X into four WPT represen- (e)
(D tations { a.j
(R)
S D
(k)
d,j This is achieved by applying expression 1 with
four suitable filter banks
{G!)
f
(
((!)
{G() Then we can compute the following
complex wavelet packet coefficients E (M) and E (R)) for each & {o...4 1}:
(R)
(k) E (k)
{ I
D (k)
I
D
R
M
D d,j
I-
-
D
For a given scale j > O, the set of (2 4)) complex matrices E (k) E (k) constitutes a re-
dundant representation of X from which the original signal can be reconstructed. DT-CWPT is oriented, and nearly shift invariant if we consider the modulus of complex coefficients. Figure illustrates the DT-CWPT resulting filters with j = 2.
Dual-tree real wavelet packet transform (DT-RWPT) By computing only the real part of the above coefficients, we get representation of X in a real tight frame. Like above, DT-RWPT an oriented transform, but does not possess the shift invariance property. This may have consequences on its predictive power, as will be seen later.
Link with Gabor filters Such as presented above, the wavelet packet transforms compute full decomposition in what Mallat (2009) calls a pseudo-local cosine basis. The resulting filters have identical window size with varying number of oscillations within these windows (see Figure I Therefore, such wavelet packets share similarities with Gabor filters. However, they offer a compet- itive advantage: the decomposition is performed efficiently using one few separable filter banks, which are fully characterized by single one-dimensional vector.
Convolutional layers Let P denote the number of samples (batch size), K (resp. L) the number of input (resp. output) channels, (M, N) the size of input feature maps and (p, v) the kernel size.
A 2D convolutional layer with fixed parameters s, d, g E No (stride, dilation factor and number of groups, respectively). weight W IR (K//))xxxxx and bias b E R&4, transforms any 4D input tensor
3

---

Under review as a conference paper at ICLR 2021
Figure @- Respectively, WPT and DT-CWPT filters for j 2, computed with Q-shift or- thogonal CMFs of length 10 (Kingsbury) 2003). The matrices have been cropped to 11 x 11. b displays 32 complex filters, alternatively represented by their real and imaginary parts. ① and
are the filters computing E and E, (R), respectively. @ Convolution kernels Watex [0, ki] in AlexNet's first layer. We used model from the Torchvision package, pretrained on ImageNet.
X E RR PxKxMxN into output tensor Y E PRPXxxM''''
such that
K/q-1
Pp, 1] = bll] +
Σ
X ko(1) + ++]+
W kt, I (dd ts.
(3)
where (2) [g/L]] [| K/q denotes the first input channel influencing the I-th output. Note that in expression Ip. I. [p. ko(() and W kk, are 2D matrices while b[l] is scalar.
Definition 1. We denote c( (W, b) the operator computing Y c(9) (W, b) X.
In this paper, we focus on AlexNet's first convolutional layer, which can be represented as a con- volution operator c (W, b), with W Ø3x64x11x11 and b e R4 The kernels W [0, k] after training with ImageNet are displayed Figure We can notice oriented oscillating patterns similar to wavelet packet filters.
3 PROPOSED MODELS
We now introduce several network architectures that are built standard AlexNet, in which the first 11 x 11 convolutional layer replaced by a succession of WPT or DT-(RC) WPT decompositions modulated by mixture weights. Each network takes input a 4D tensor X R Px3x224x 224 set of P images with three input channels (RGB images).
WTT
i.e.,
3.1 WPT MODULE
This module computes two successive WPT decompositions T for every input channel. Each step {0, 1} is implemented as strided convolution operator c(a) (W,, 0) (see Definition where W, contains the fixed low- and high-pass filters. In this configuration, each input channel convolved with its own set of filters.
More precisely, we have (s 2), (d and (gj K2), where K2 (3 43) denotes the number of input channels. W {IX(X))))XxI is such that for all & E {0.. - 1} and (0..3),
W, [0, 4h + G()) The output, denoted R 18x8x xX" such that
12) D = 2.1 (W1, 0) (c (Wo. 0) x
(4)
Once this done, we need to modulate the importance of each wavelet packet. Moreover, the number of output channels must be equal to 64 as in standard AlexNet, and every output channel

---

Under review as a conference paper at ICLR 2021
must be influenced by each input RGB channel. This is achieved with 1 x convolutional layer (Lin et al. 2014 Szegedy et al. 2015) placed after the WPT decomposition. Note that this approach was also chosen Ulinyy et al. (2019) in what they call harmonic block.
Therefore, the final output, denoted Y wpt E ER" xXx66x56556 such that
(1) Y wpt 1, (5,1 mix: bmix)- D
where Wmix E 18X6611xX1 and mmix E R&1 are freely trained. A schematic representation of the
WPT module can be found in Figure 2 The orange ("FB", a.k.a., filter bank) and green ("Conv")
2
layers compute expressions (4) and respectively.
Number of trainable parameters A WPT module has mixx] + |iix 3, 136 trainable pa- rameters, versus 23, 296 for the first convolutional layer in a standard AlexNet.
3.2 DT-(RC) MODULES
In DT-RWPT and DT-CWPT modules, respectively and four suitable filter banks are used on each input channel. The outputs, denoted ER E IR Px96x N' x' and Ec R Px 192x x have the following structure. For any sample p E {0.. (0..P. P..... -1,
Re E
[p)
D [a(p]
Da (p)
Re E (p) Rs E [p]
[plp] D [p]
Re E
[p]
D [pl
Da [pl
ER [p]
Eclp]
Da.(p) [pl Da(p)
Im [p]
Delp)
D
Im
[p]
Dr(p) Db(p)
(6)
where D., Dp, De and Da E PP458XX X are computed similarly to 4
Expression (6) is tensor formulation of where the real and imaginary parts of the complex coefficients are stored separately. As for WPT computed in (4), both DT-RWPT and DT-CWPT can be expressed as succession of CNN-style convolution operators. This requires few technicalities
that are provided in Appendix A.4
convolutional layer after the wavelet packet decompositions. The final outputs, denoted Y dwpt and Y dt-CCwpt IR Pxx1xX'' XX'" are such that
Again, placed
dt-&wwpt = c (W (W uixx! b'mix) ERi
dtt-Cwpt = c (W" mixx 6" Ec
(7)
where W W'ix E 1 W mix E R 192x64xxx1 b' b'nxx and b" E R are freely trained. A schematic representation of both modules be found in Figure 23 The blue (*T" and "+") and green ("Conv") layers compute expressions (6) and 77. respectively.
Number of trainable parameters DT-EWPT and DT-CWPT modules have 6, 208 and 12, 352 trainable parameters, respectively (23, 296 in a standard AlexNet). We will see in Section how these numbers can be further decreased without degrading the performance of the network.
3.3 KERNEL VISUALIZATION
WPT and DT-(RC)WPT modules are designed as succession of multi-channel convolutional lay- The following proposition states that such cascading layers can be expressed single CNN- style convolution operator. It provides an explicit formulation the resulting hyperparameters ie., stride, dilation factor and number of groups describing input-output channel connections and weight tensor. It takes advantage the well-known result that two successive convolutions can be written another convolution with wider kernel.
Let c!!) (W, b), denote an initial convolution operator, with W E TR KXLxxxx We consider a second operator c/ (V, a), with V e x and a E R (we assume that both L and L' divisible by g)
5

---

Under review as a conference paper at ICLR 2021
224. 224.
224. 224, 3
224. 224. 3
224. 234.
FB
FB a
FB d
FB a
FB d
c
b
112. 112.
112. 112.
112, 112,,
FB a
FB d
FB a
FB d
FB
FB b
Conv (11, 11) 14
FB
56, 56,
Real part
Stack
Imaginary part
56,
Conv (1. 1)
CocV (1. 1)
Conv (1, 1)
55, 58,
56, 36,
56, 56,
86, 56, 61
Figure 2: ① First layer of AlexNet; @ WPT module; 2 DT-RWPT module; 4 DT-CWPT module. Only the green layers (Conv) have trainable parameters. The numbers between each layer indicate the height, width and depth (number of channels) of the current feature map tensor.
Proposition The composition of two strided CNN-style convolution operators such as introduced by Definition 7 can be written as another strided CNN-style convolution operator:
(()) (CL.) (V, a) (,1(W,,)))) b)) Cst. (W', b'),
with s' = st (stride), d' = (dilation) and q' 1 (number of groups). Moreover, the resulting weight W' is computed using a dilated CNN-style convolution operator:
W'
claw) Sw,
(V, 0) W).,W
with (sw = 1), (dw s) and (aw = q)- We also have b' a f(V, b), where f is such that f(V, 0) An explicit formulation of is given the Appendix A.6
Remark. Proposition I is valid only if the matrices are infinitely extended beyond their limits we either get an infinite sequence with finite support (zero padding), a -N-periodic signal (periodic padding) or a 2.N -.--porioiic signal (symmetric padding). In practice, the amount of padding at each layer must be carefully chosen to avoid distortion effects at the edges of feature maps, as done in our implementation.
Proposition I whose proof given Appendix A.6 shows that the wavelet packet modules com- pute
wpt C. (W wpt, Umix)) X
(Wwpt,
(10)
(1) C4 (Wat-Rwpt: b'nis) X
(1) Y Ydt-Cwpt C4,1 (W Wa-CCwpt 6" b"ix) X
Y dt-EEwpt
(11)
where Wwpt, Wpt: Wat-Cwpt (3x 2) (3p-22 are obtained from 9). As reminder, denotes the size of the CME By means of comparison, AlexNet's first layer computes
Y alex ((1) (Walexx alex; balex) X
(12)
where Walex R3x61x11x11 and balex E 64 Therefore, all modules are represented as convolu- tion operators with stride 4, mapping input channels to 64 output channels. Regarding kernel size, it is bigger than 11 as long as μ > 5; However, most energy is concentrated in region the size of which is similar to AlexNet kernels.
A visualization of these kernels after training is given in Figure Training details are provided in Section For the sake of visual comparison, we only displayed center patches size 11 x 11 from the original matrices it turns out that in all our models, between 97% and 99% of their energy (i.e., the squared L - is concentrated in these cropped regions. We point out that computing the resulting kernels is used for analysis purpose, but should not be involved in the training process.

---

Under review as a conference paper at ICLR 2021
Figure 3: From left to right: (W wpt (0, k]], Wtpt [0, k]], aWtt- [0, k) ko after training with ImageNet ILSVRC2012. All modules are implemented with Q-shift filter (μ = 10). Whereas the WPT module mainly extracts horizontal and vertical features, many more orientations arise from the dual-tree modules. We can also notice the tow-pass filters, which appear as color blobs. The resemblance with AlexNet kernels (see Figure I is prominent.
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
Wavelet filters For the experiments we used PyTorch / CUDA implementation. Our wavelet packet modules were designed with a Q-shift orthogonal filter of length 10 (Kingsbury (2003), which approximately meets the half-sample-shift condition required for the dual-tree transforms. For the sake of setting consistency, we also used this filter for conventional WPT.
Datasets Our models were trained and evaluated on ImageNet ILSVRC2012 dataset (Russakovsky al. 2015). Since the online evaluation server is longer available, we aside 100, 000 images from the training set 100 per class in order to create a validation set. We used this subset to compute accuracy rate along the training phase. As for the validation set provided by ImageNet, we turned it into test set which our trained models were evaluated.
To assess generalization performance of our models, we also finetuned them on PASCAL VOC 2012 (Everingham et al.) 2015) and COCO 2014 4 (Lin et al. (2015) datasets, on the multilabel classification task. For this we initialized the networks with the parameters previously obtained with ImageNet and replaced the last fully-connected layer by layer containing the desired number of outputs. Since again we didn't 't have access to the ground truth for the "official" test sets, we split each validation set in two roughly equal parts. We then used the first part for validation and the second for testing.
For each dataset, the models were trained on a single GPU. The training pro- cedure was inspired from many ILSVRC papers (Krizhevsky et al. 2012, SSmonyya & Zisserman 2015 Szegedy et al., 2015 He et al. 2016). More precisely, it was carried out by optimizing the cross-entropy loss with stochastic gradient descent. For this we fed the network with random batches of 256 images, until reached 100 cycles through the whole training set (100 epochs, i.c., 461.4K iterations for ImageNet). The momentum was set to 0.9 and weight decay to 5 10-4. As for the learning rate, it was initially set to 10-2 and then decreased by factor of 10 every 25 epochs.
Training details
To reduce overfitting, we followed the data augmentation procedure used in Inception networks (Szegedy al. 2015).. The images are first normalized specified mean and standard deviation for each RGB channel. Then they are randomly flipped and cropped from 8% to 100% of their original sizes, with random aspect ratio varying from 3. before being resized to 224 224 using a bilinear interpolation.
Model evaluation The test phase was carried out following Krizhevsky et al. (2012). Namely, predictions are made over 10 patches extracted from each input image, and the softmax output vectors are then averaged to get the overall prediction. We used top-1-5 accuracy rates (ImageNet) and average precision (multilabel tasks) (Everingham et al. 2015) evaluation metrics.

---

Under review as a conference paper at ICLR 2021
Frozon Alextle:
Frogen Alcxlett
WPT AlexNet
WIPT AlexNet
Alechast
AlexNuE
Cross-eiropy
colidaaion
50000
200000
apaccts
Figure 4: Evolution of the loss function (left) and top-1 validation error (right) during the first 65 training epochs. Validation has been performed by simply resizing the smaller edge of each image to 224 and extracting a single patch of size 224 x 224 at the center.
Table I: Error rates on our custom validation and test sets.
ImageNet
VOC
COCO
Model
Nb params
Top-1 (test)
Top-5 (test)
Top-5 (val)
Average error (test)
WPT AlexNet
3.1K
45.5%
22.9%
20.2%
25.6%
46.0%
DT-RWPT AlexNet
6.2K
43.2%
20.7%
18.2%
23.6%
44.1%
DT-CWPT AlexNet
12.1K
12.3%
20.2%
17.5%
23.1%
43.4%
Standard AlexNet
23.3K
42.2%
20.0%
17.5%
22.9%
43.4%
Frozen AlexNet
None
51.1%
27.8%
24.8%
31.2%
51.3%
Comparison with existing models Our models were compared with standard AlexNet that we trained according to the same procedure. In addition, we wanted isolate the contribution of wavelet packet modules to the global predictive power. To achieve that we trained an AlexNet convolutional layer was frozen to initial parameters.
which the first
4.2 RESULTS AND DISCUSSION
The evolution of the loss function and validation error along training with ImageNet is shown in Figure 4 Similar graphs be found in Appendix A.3 for VOC and COCO datasets. Besides, classification performance of our trained models are displayed in Table [] For multilabel tasks, we have defined the average error by E II [0, 1J. where II denotes the average precision.
The DT-CWPT models almost reach standard AlexNet's accuracy, with twice less parameters in the first layer. While WPT and DT-RWWP models achieve lower performances, they still much higher than the frozen version of AlexNet. This result suggests that the predictive power is partly accountable the wavelet packet modules themselves. and not entirely the following layers. Besides, the good results we obtained on multilabel classification tasks assert the generalizability of our models.
By looking at Figure it is easy to explain why conventional WPT has the lowest accuracy of all our models. We identified two main reasons. (1) The filter design makes it impossible extract oriented features that are neither horizontal nor vertical. Instead, it yields checkerboard patterns, which are useful to catch the remaining information and ensure perfect reconstruction, but fail process geometric image features like ridges and edges, pointed out by SSlesnick et al.) (2005). (2) There is no medium- frruuumyyyyyueccyy feature extractor like in AlexNet kernels black-and-wor and-white patches side by side. Such patterns imaginary parts of the complex low- low-pass filters separately, we actually get one low-pass and one oriented band-pase pandd-passsss filter see Figure
however be found in dual-tree kernels. If consider the real and
Regarding the DT-RWPT model, it performs better than WPT but does not reach the accuracy of DT-CWPT or standard AlexNet, despite generating oriented filters. Intuitively, DT-CWPT is twice
8

---

Under review as a conference paper at ICLR 2021
more redundant than DT-RWPT, and thus is more likely extract relevant features for image clas- sification. We propose here a more specific interpretation. In Section 2 we mentioned the shift invariance property of DT-CWPT, that neither WPT nor DT-RWPT possess. By applying slight shift to an image, great disturbances can indeed be observed in the matrices of real coefficients (Selesnick et al. 2005). Important features extracted from one image thus disappear from the other. On the other hand however, the modulus of complex wavelet packet coefficients is nearly shift invariant, meaning that their value is smoothly transferred toward the neighboring pixels in the shift direction. Therefore, any loss of information in their real part is recovered in their imaginary part. The DT-CWPT module is capable to extract similar features from two shifted images, which could explain its superior performances. We tested the robustness of our models with respect to small shifts and obtained results that support our hypothesis. More details can be found in Appendix A..
The source code used for our experiments will be published shortly after paper acceptance. We will also provide notebooks for the sake of replicability.
CONCLUSION AND FUTURE WORK
Following an epoch of frantic race toward classification performance, research is more and more focused on understanding learning mechanisms in CNNs. In this perspective, we proposed chitecture in which standard convolutional layers are replaced by dual-tree wavelet packet feature extractors. Our experiments show that such networks can compete with conventional models, pro- viding sparse description of their behavior.
The DT-CWPT module contains twice less trainable parameters than standard AlexNet's first layer. Future research could be held to further increase the sparsity of our models. (1) Some filters gen- erally extract less information than others, in that the corresponding feature map's energy is much lower. By discarding these feature maps before computing 1 x 1 convolutions, we could reduce the number of parameters by the same amount. More details about energy distribution is given in Ap- pendix A.2 (2) Feature map combinations may not be equally worth. We could constrain the 1 x convolutional layer by preventing some scales or orientations from wiring together, or by separating the real and imaginary coefficients.
So far we trained our models with a predefined CMF, but other filters may provide better extraction properties due higher frequency localization number of vanishing moments. One way to address this question could be to let the network learn the optimal filter with a proper regularizer in the loss function. This will be addressed in future work.
We tested our framework on the first layer of AlexNet, because introducing wavelet packet trans- form into it is quite straightforward. Nevertheless, the phenomenon of oscillating patterns does restrict to this particular model, nor is it specific to the sole task of image classification. However, extending our models sired hyperparameters. This will be tackled in future work, in which we will also benchmark our results with other wavelet CNN approaches.
wider range of architectures must be handled carefully match the de-
REFERENCES
Ilker Bayram and Ivan W. Selesnick. On the dual-tree complex wavelet packet and M-band trans- forms. IEEE Transactions on Signal Processing. 2008
Y-Lan Boureau, Francis Bach, Yann LeCun, and Jean Ponce. Learning mid-level features for recog- nition. In 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 2559-25666 2010.
Joan Bruna and Stéphane Mallat. Invariant Scattering Convolution Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3578):1872-1886, 2013.
Shuo- Yiin Chang and Nelson Morgan. Robust CNN-based speech recognition with Gabor filter kernels. In Fifteenth Annual Conference of the International Speech Communication Association, 2014.

---

Under review as a conference paper at ICLR 2021
Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes Challenge: A Retrospective. International Journal of Computer Vision, 111(1):98-133) 136, 2015. ISSN 1573-1405.
Shin Fujieda, Kohei Takayama, and Toshiya Hachisuka. Wavelet Convolutional Neural Networks for Texture Classification. 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016.
Ke Huang and Selin Aviyente. Wavelet feature selection for image classification. IEEE Transactions on Image Processing, 2008.
Nick Kingsbury. Complex wavelets for shift invariant analysis and filtering of signals. Applied and computational harmonic analysis, 10(3):234-253, 2001.
Nick Kingsbury. Design of Q-shift complex wavelets for image processing using frequency domain energy minimization. In Proceedings International Conference on Image Processing, volume 1, pp. I-1013, 2003.
Alex Krizhevsky, lya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks, In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Andrew Laine and Jian Fan. Texture Classification by Wavelet Packet Signatures, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1993.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 2015.
Min Lin, Qiang Chen, and Shuicheng Yan. Network In Network. arXiv: [cs], 2014.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. arXiv:1405.031 [cs], 2015.
Hongya Lu, Haifeng Wang. Qianqian Zhang, Dachan Won, and Sang Won Yoon. A Dual-Tree Complex Wavelet Transform Based Convolutional Neural Network for Human Thyroid Medical Image Segmentation. In IEEE International Conference on Healthcare Informatics (ICHI), pp- 191-198, 2018.
Shangzhen Luan, Chen Chen, Baochang Zhang, Jungong Han, and Jianzhuang Liu. Gabor Convo- lutional Networks. IEEE Transactions on Image Processing, 27(9)4357-4436 2018.
Stéphane Mallat. A Wavelet Tour of Signal Processing The Sparse Way. Elsevier/Academic Press, 2009.
Edouard Oyallon, Eugene Belilovsky, Sergey Zagoruyko, and Michal Valko. Compressing the Input for CNNs with the First-Order Scattering Transform. Proceedings of the European Conference Computer Vision (ECCV), pp. 301-316, 2018
Stefan Pittner and Sagar V. Kamarthi. Feature extraction from wavelet coefficients for pattern recog- nition tasks. IEEE Transactions on pattern analysis and machine intelligence, 21(11)83-883 1999.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-25) 2015.
Syed Shakib Sarwar, Priyadarshini Panda, and Kaushik Roy. Gabor filter assisted energy efficient fast learning Convolutional Neural Networks. In 2017 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1-6, 2017.
10

---

Under review as a conference paper at ICLR 2021
Ivan W. Selesnick, Richard Baraniuk, and Nick Kingsbury. The dual-tree complex wavelet trans- form. IEEE Signal Processing Magazine, 12 2005.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations, 2015.
Amarjot Singh and Nick Kingsbury. Dual-Tree wavelet scattering network with parametric log transformation for object classification. In ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing- Proceedings, 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Computer Sociery Conference on Computer Vision and Pattern Recog- nition, 2015.
Matej Ulieny, Vladimir A. Krylov, and Rozenn Dahyot. Harmonic Networks for Image Classifica- tion. In British Machine Vision Conference, pp. 202, 2019.
Travis Williams and Robert Li. Advanced Image Classification Using Wavelets and Convolutional Neural Networks. In 2016 15th IEEE International Conference on Machine Learning and Appli- cations (ICMLA), pp. 233-239, 2016.
Travis Williams and Robert Li. Wavelet Pooling for Convolutional Neural Networks. In Interna- fional Conference on Learning Representations, 2018.
Gary G. Yen. Wavelet packet feature extraction for vibration monitoring, IEEE Transactions on Industrial Electronics, 2000.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, pp. 3320-3328, 2014.
11

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 ROBUSTNESS OF OUR MODELS WITH RESPECT TO SMALL SHIFTS
To assess the robustness of our models with respect to small shifts, we compared the network outputs between reference image and cight shifted versions along each axis, over our custom ImageNet test set (50, 000 images). To do so, the Kullback-Leibler divergence is computed between each pair of softmax activation vectors (E IR 1000) after forward-propagatio through the network. An illustration of the results can be found in Figure
0.08
Mean KL divergence
Mean KL divergeecee
0.06
0.04
0.04
Standard AlexNet
0.02
Standard AlexNet
WPT AlexNet
AlexNet
AlexNet
DT-RWPT
0.00
DT-CWPT AlexNet
0.00
DT-CWPT AlexNet
Shift along axis (nb of pixels)
Shift along axis (nbb of pixels)
Figure 5: Shift-robustness of our models, compared to standard AlexNet. The Kullback-Leibi di-
vergence is computed between output vectors and averaged over the whole dataset (50, 000 images).
When the input image is shifted by 4 pixels, the output of the first layer is strictly shifted by one pixel. The first layer is therefore invariant to a 4- 4-pixel shift. Consequently, any divergence between outputs should be due either to edge effects or the action of deeper layers. Likewise, when the shift is equal to 8, the invariance applies to the two first layers. However, when the shift is not a multiple of
observe bigger discrepancies which depend the chosen model.
The sensitivity to small shifts seems to increase with the network's predictive power. On the other hand, observe higher discrepancies for WPT AlexNet and DT-REWPT AlexNet, compared to DT- CWPT AlexNet. This is in agreement with our hypothesis that the nearly shift-invariance property of DT-CWPT is to some extent conserved across the whole network, and therefore brings competitive advantage regarding predictive power. However, DT-CWPT AlexNet fails to reach the shift-robustness of standard AlexNet, suggesting that further improvements could be brought to our models (see Section 5).
A.2 ENERGY DISTRIBUTION OVER FEATURE MAPS
Figure 6 displays the mean energies of the 30 feature maps of high-frequency DTTCWPP coeffi cients, computed over custom ImageNet test set. As see, the energy distribution is very unbalanced over the different filters.
A.3 TRAINING AND VALIDATION CURVES FOR VOC AND COCO DATASETS
The evolution of the loss function and validation error along training for multilabel tasks is shown in Figure 7 The graphs share similarities with Figure These experiments suggest that DT-CWPT AlexNet has good generalization performance on other recognition tasks.
We notice the erratic aspect of the validation curves during the 25 first epochs. This may be due to poor learning rate initial setting. After decreasing this parameter, the validation errors become more stable.
12

---

Under review as a conference paper at ICLR 2021
Mean feature map energy
8000
6000
4000
2000
0
5
10
15 Filters
20
25
30
Figure 6: Mean energies of the 30 feature maps of high-frequency DT-CWPT coeffiiettt over our custom ImageNet test set.
Standand Alextlt Frozen AlexNet WPT Alexhic
0.5
Standard Alexket
0.5
0.50
WPT Alextet
JUNPI Aloxtatt
4WMP Alexhat
sveraaee ether
3.4%
0.8
0.5
0.35
0.30
0.3
0.25
1500
Kiumber Eeraaioos
Mlrr of
8.5
0.75
WIT Nestte IT-SINPT Alestiet
WTT Aleatte
DT-GWWP Nevket
DT-CWPT
Averoge error
Cosz-entroyy
3.55
0.60
0.45
0.45
Number Herctionns
Number opachs
Figure 7: Evolution of the loss function (left) and average error (right) over VOC and COCO vali- dation sets (top and bottom, respectively).
13

---

Under review as a conference paper at ICLR 2021
A.4 IMPLEMENTATION THE DT- WRC) MODULES
In this section we show that the dual-tree transforms can be written as a succession of CNN-style convolution operators. We focus DT-CWPT but it be easily adapted to DT-RWPT.
The first step is to duplicate each input image four times (one for each filter bank). Given an input tensor X IR Px3x224x224 this operation can be written CNN-style x convolution operator:
Xo ((1)
dVdul, 0) X
(13)
where Xo E P2X12X2242224 and dupp = (1 I 1) E R with E gax3xlxi such
1f
that [k, 1, 0,
0 otherwise.
Then each level of filter bank decomposition can be summarized into a single CNN-style convolution
operator c/, (Wj, 0), with (s 2). (d 1) and (gj = Kj). where Kj = -33
=(3 (344))) denotes
the number of input channels. For any j {0. 1), the weight tensor W, E IR ( x has the following structure:
Wa.j 00]
Wb.5 [0]
W
(14)
W 0
Wa [0]
where Wa.j. Wb.j. We,5 and Wa.j are built similarly to the WPT module, using filter banks (
f
{ p (( and ( respectively. By denoting D the output this stage, we get
D = c (8)) (W1, 0) (c). (Wo,
(Wo, 0) xo)
(15)
Remark. Note that we have, for all sample p e {0. P 1}.
D [p]
[P(p] Dp
D(p] =
(16)
Da(p]
Finally, expression o is expressed as CNN-style convolution operator:
(1) Eg C (V combine: 0).D, D,
(17)
where Veombine R 192x 192x192x1x
has the following structure:
0
0
0
1
0
combine
1
0
0
(18)
0
-1
0
with
E IR xxx such that l[ke I, 0, 0] 1 if :
otherwise;
O R 48x such that Ojk, 1, 0, 0] 0 for all k, 1 E 47..
Note that the two last dimensions of W Weombine have size x 1; the "convolution" kernels are thus reduced to singleton matrices.
Therefore, expressions B (5) and provide a description of DT-CWPT as a succession of
CNN-style convolution operators.
14

---

Under review as a conference paper at ICLR 2021
A.5 AN ALGORITHM TO COMPUTE THE RESULTING WEIGHT AND BIAS
Proposition I provides an iterative algorithm to compute the weight and bias resulting from suc- cession of CNN-sttle convolution operators. special care must be paid initialization. The number of groups in the first convolution operator must indeed be equal to 1. In order to meet this requirement, we introduce an identity operator C, (I, 0), where IR xxxxxx is defined by
[1 kf
0, 10 otherwise.
Proposition 2 (Identity operator), Let K, L E N' and t, q E No such that both K and L are divisible by q. For all V (2(K9))xxxxx and all a E Rt,
cV a) = (G. a)o
(19)
Proof. It can be easily proven that for all X BPXXXNXN c. (1, 0) x x.
Therefore Proposition 1 can be used on expression 19 in order to initialize the algorithm. The details are given in Algorithm
Algorithm Composition of convolution operators
Require: K = Lo {number of input channels}
Require: {(L1. f1, 91. W1. by)..... (Ln. tns Ir, Wn. bn)} flist of output channels, strides, groups, weights and bias}
Ensure: E (1.. R), both - and L, are divisible by dr
Ensure: Vr (1.. R}. (W,) = [ -(
Tor Pr vx) and |6,| Lr
W E XxxxII (identity weight}
b < 0 (initial bias} + I (initial stride}
for E {1.. RR} do (Wr, 0) W (resulting weight)
W + c/., clco))))w..)
b - b, + (p(())) [.. b, S.W.) (e s..)))=gfo.e))) fresulting bias}
f- x tr [new stride]
end for
W + W {flip weight tensor along last dimensions)
return (8. W, b) (resulting stride, weight and bias}
A.6 PROOF OF PROPOSITION 1
Proof. Let X E R denote an input tensor. Let Y E RPxLxMXM and Y' PXL MM denote the outputs of the convolution operators, such that
Y= c C (W, b) X and Y'= (V, a) c (W, b)) X
cV a) (c.) b) x
(20)
CV a) Y
Let p E (0.. P 13. By using 3
and 20p. we have, for all I' E (0.. L' 1}.
L/g-1 1
l'q
L
11 -
+
Σ
p.
+1 l
V
it,
(21)
L'
q
1=0
15

---

Under review as a conference paper at ICLR 2021
and, for all I E {0.. L-.L-11 1}.
K
Y(p, = b[4
+
X(p, k) W[kk Us.
(22)
k=0
The next steps require the two following lemmas: Lemma 1. For all 2D matrices U, V, and all b E R,
(b+U)+V= b > Vm,.n +U+V)
(23)
Lemma 2. For all 2D matrices U, V, and all integers s, E N (IU ( V) 1t= Lt= (U (V 1 8)) (st).
(24)
where, as a reminder, (V 1 8) denotes the s-dilated matrix.
Then, by plugging
22 into 21 and by using Lemma 1 we get
L/g-1
=
b
l'q
L
1
Σ Vil, 1' m, nl
Evi,
L'
q
(25)
m,n
L/q-1
Σ
[p,
k]
W
k.
l'g
L +1
s* vil.n]
b2:
L'
and finally, by reversing the 2 sums and using Lemma get
W K.Pl
K-1
L/g-1
l'q
-1p.
+
X X(p, k]
W
k
+
* VI,
f
4 (st)
L'
km
I=
(26)
K-1
= b +
X(p, k] * ('),(']) 1 (st)
k=
Hence
Y' = c(1) (st), (W', B') X
(27)
for all which proves
By applying the definition of a CNN-style convolution operator
3 to the definition of W [k,'') in
expression 26 get
w' =
(()) (V, (V, 0) W
(28)
which proves
9
Note that the expression of b' b'(')] defined in 25 can be rewritten in a more concise way:
b' [1'] = al'' ( S.V).
(29)
where
{p(1)
denotes partition of even-size slices of b. For any ( 1}.
11(0.0-1)
YK (y + 1)K 1
p(9)
(30)
16

---

Under review as a conference paper at ICLR 2021
S.W RK/y denotes the vector such that for any & {0... K/q - 1/.
(s:W) = Σ
Wik, L, m, n]
(31)
m,,
Let's denote f(V, b) R² such that f(V, 6).7 = (PlWo/znb, b, S.W). Then we get f(V, 0) O, stated in Proposition
Remark. Operators p( and S have the advantage of being efficiently computed with libraries such as PyTorch or even NumPy.
A.7 ILLUSTRATION OF WPT AND DT-CWPT
Figures and illustrate WPT and DT-CWPT, respectively. We chose an image from our custom ImageNet test set (one channel only), resized and cropped to 224 224 pixels. We can notice that specific orientations tend to be selected by specific filters, especially in the dual-tree transform. Moreover, some filters seem to extract features of higher energy than others.
17

---

Under review as a conference paper at ICLR 2021
Dj (8)
(1)
D2
D
(2(4)
Dy (
.2)
(3)
D2 (6)
PPP
(3)
D2
D2200
121
03033
D2000)
D
D_114)
D
Figure 8: Left: original image from ImageNet2012 (j 0). Middle: WPT with j 1. Right: WPT with = 2. At each step j. the feature maps of wavelet packet coefficients D. are further decomposed into four smaller submatrices D j+1 ((4++) for E (0..3). according to 1 To avoid
overexposure, scaling coefficients D have been set to 0 (top-left images).
Figure 9: Modulus of the complex DT-CWPT coefficients, computed with 2:
E (k)
(left) and
E (right), for k E {O.. 15}. Numbering follows the order of Figure 8 To avoid overexposure, scaling coefficients E (0) and E, (0) have been set to 0 (top-left images).
18


Reviews:
[{"review_id": "ol7JfpTad1g", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose a scheme for combining the trusted mathematical properties of Dual Tree Wavelet Packets with CNN-style feature extraction. Specifically, they learn simple functions of wavelet kernel outputs, rather than learning kernels themselves from scratch. The fundamental gains are a significant drop in the number of parameters, while retaining feature expressiveness and intuition.\n\nClarity:\nThe paper clarity is significantly above average.\n\nQuality:\nThe paper is very clear and concise, but it might be better if it compared across other nets besides AlexNet. For example, how does it compare to nets with feature extractors in the same order of magnitude (3k parameters), vs much bigger (alexnet: 25k).\n\nOriginality/Significance:\nI am not extremely familiar with literature related to this idea. However, I think constraining CNN filters in this way is an important area of research.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jDfpY8KDRcn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a modification to the prevalent CNN architecture that leverages filter banks based on wavelets with the motivation to reduce trainable parameters and improve interpretability. More specifically, the first CNN layer of the AlexNet architecture is replaced with a wavelet packet transform (WPT). The authors show the wavelet packet transform can be written as a series of convolution operation and visually compare the filters of the trained AlexNet to the WPT filters. Experiments on ImageNet show the proposed method can match the performance of AlexNet on ImageNet with fewer parameters.\n\nPros:\nThis work tackles in a focused way the interpretability and formalism of deep neural networks, specifically CNN, by grounding the first layers of the network in the frame work of the wavelet transform.\n\nThe similarities shown in Figure 1 between trained AlexNet kernels and kernels from the proposed method are interesting\n\nThe reduction in learned parameters for the network under the proposed approach is a nice feature, and shows that stronger priors can guide learning.\n\nCons:\nA crucial limitation of the proposed approach seems to be its application only to the first layer of the CNN. It would be interesting to apply the method to multiple layers of the network or in the extreme case, a network based solely on the WPT.\n\nAnother limitation of this work is that the proposed method is not applied to other popular and performant CNN architectures, e.g. ResNet (He et al. 2015), and is only applied to the relatively older AlexNet model.\n\nTo be more convincing, the method must be demonstrated on more datasets or tasks. For example, other image recognition datasets or other tasks (such as ASR). Results on one CNN architecture on one dataset does not fully demonstrate the generalizability of the method.\n\nThe authors show the proposed method reduces the number of learned parameters, however does the method reduce the total number of parameters? Is the inference time of the network improved/degraded with respect to the CNN baseline?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VhZTM_z4jnO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a variation of the popular AlexNet architecture, where the first convolutional layer is replaced with a wavelet packet decomposition. This is motivated by the fact that the first-layer convolutional kernels look very similar to wavelet filters in that they mostly extract oriented edges and smooth gradients. The wavelet packet coefficients are then weighted using a single mixing layer implemented as a 1×1 convolution. The resulting module (wavelet packet decomposition plus 1×1 convolution) has a smaller number of learnable parameters, but achieves close to the same performance as the standard AlexNet on the ImageNet ILSVRC2012 dataset.\n\nThe paper is well laid-out and the important technical concepts are explained in a clear and concise manner. The various wavelet packet modules used to replace the first layer of AlexNet are well described and their differences clear. The same goes for the experimental section and the evaluation of the results. On the theoretical side, I am not sure that the result on page 6 qualifies as “theorem” – the fact that two successive convolutions can be written as another convolution with a wider kernel is well-known in the signal processing literature. While the experimental results are impressive, more could also be made out of the analysis. I also do not see any evidence that the authors intend to publish the code for this experiment. Despite these issues, I think the results are interesting enough that the paper be accepted for publication in the proceedings.\n\nAs discussed in the paper, there is a great need for theory explaining the performance of deep neural networks. This work is a step in that direction, reducing the number of learnable components and replacing them with fixed representations (here: wavelet packet decompositions) and achieving the same performance.\n\nRegarding the theorem, again, this is not a new result, fundamentally, and I think this should be made clear. That being said, a potential subtlety here that needs to be addressed, is the behavior at the edge: for large enough kernels, unless the boundary extension is periodic, the composition of two convolution kernels is not necessarily another convolution. This potential issue is not discussed in the main text nor in the proof of the theorem.\n\nCurrently, the main push of the analysis is that the first-layer kernels can be replaced by similar-looking wavelet packet kernels. While important, similar results have been established before (by Oyallon et al., 2018, for example), and it is not clear what the wavelet packet approach brings to the table. Do these filters have certain properties that make them easier to analyze? Is the resulting reduction in number of learnable parameters greater?\n\nSection 4.3 is also a bit odd. It is mostly speculation about how the number of parameters could be reduced further. Since the outlined scheme is rather simple, why have the authors not tried it? Otherwise, I do not quite understand why this section is included. Similarly, the authors speculate that the dual-tree real wavelet packets perform worse than the complex variant due to higher sensitivity to image shifts. Whether this is the case can be easily tested experimentally by shifting the images in the test set. Have the authors explored validating the hypothesis in this manner?\n\nThere are also a few small problems in the main text:\n– p. 3, in Definition 1, it is not clear how the operator is defined. A reference to eq. (3) should suffice.\n– p. 4, “every input channels” should be “every input channel”.\n– p. 7, “predicting power” should probably be “predictive power” here and throughout.\n– p. 8, “extract lesser information” should be “extract less information”.\n– p. 8, “does not restricts” should be “does not restrict”.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6r8MdHNbEED", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To improve the understanding of CNNs, this paper proposes a method to constrain the behavior of AlexNet by replacing its first layer with a module based on wavelet packet decompositions. Three variants of wavelet decompositions are evaluated, including the separable wavelet packet transform and the 2D dual-tree real and complex wavelet packet transforms. A visualization algorithm is implemented to compare the extracted features with that of AlexNet. Results on an image classification task show that the accuracy of standard AlexNet can be achieved with less trainable parameters. \n\nPros:\n+ It is interesting to see the combined convolutional kernels, as shown in figure 3, resembles that of AlexNet. \n+ The paper presents some intuitive explanations on some aspects of the results, including (1) why and how the proposed DT-CWPT module can reduce the number of trainable parameters, with the accuracy rate maintained compared with standard AlexNet, and (2) why the three variants differ in accuracy. \n\nCons:\n- The insight provided by this paper is vague, and it is not so clear how this can improve our understanding of CNNs (which was claimed as one of the contributions). Furthermore, what is the benefit of replacing the first layer of AlexNet with pre-designed non-trainable convolutions? Do we have better adversarial robustness, out-of-distribution transfer, etc.? \n- The experimental results of the paper do not seem very convincing. \n(1) Despite the reasons stated in the first paragraph of page 2, AlexNet is not SOTA nor near SOTA, and the adaption of such a benchmark does not seem to provide persuasive results. \n(2) In the proposed networks, only the first layer of AlexNet is replaced, but most of the parameters are in the deeper layers, especially the FC layers. It is hard to see the advantage of such a reduction of trainable parameters. \n- Lack of experiments. \n(1) Only one dataset is used, and we do not know how the filters would be affected when applied to a different dataset. \n(2) Do other choices of CMFs generate different results? \n(3) In the introduction, the paper refers to some previous work on replacing freely-trained CNN layers with more constrained structures. What is the advantage of the proposed work over the existing ones? \n\nFurther Questions:\n* In figure 3, only filters of the red channel (i.e., $W[0,\\cdot]$) are shown. But readers could also be interested in $W[i,\\cdot]$ for $i=1,2$. Maybe a colored visualization of the filters is better. \n* Still figure 3, how exactly is the filter cropping done, and how can we be convinced that the cropped parts of the matrices are negligible, e.g., with small Lp norms? \n* Appendix, algorithm 1: Does $\\langle W_r\\rangle$ mean the shape of the tensor $W_r$? \n* Section 3.1: The number $N'$ in the dimension of $D$ becomes 56 in  $Y$. Does the full-convolution (as defined in section 2) guarantee $N'=56$, or is there a cropping procedure? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is motivated by the observed similarity between learned filters at the low layers of a convolutional neural network and oriented Gabor filters. It proposes to replace the lower layers with dual tree wavelet packet transforms, which yield fixed oriented frequency-selective features. Instead of learning filters from scratch, it proposes to learn only a scalar importance for each of these features, reducing the number of learned parameters. Experiments with the AlexNet architecture on ImageNet indicate that this modification does not reduce performance, but does significantly reduce the number of parameters. The paper argues that this modification also improves the interpretability (and in the case of complex dual tree wavelets, potentially the invariance properties) of the low-level features. 

Pros and cons:

[+/-] As the paper clearly argues, replacing learned filters with wavelet packet transforms improves the interpretability of the low layers of a convolutional network. While other works have pursued similar ideas, limiting the conceptual novelty, at a technical level this is the first work to use the dual tree complex wavelet transform for this purpose. The DT-CWT may have mathematical advantages. The paper and rebuttal argue that it it is conceptually cleaner (“sparser”, since the transform is generated by a single filter) although there may not be a greater reduction in the number of trainable parameters.

[+] The additional per-channel weights are redundant in terms of the representation capacity of the network, but may effectively introduce sparse regularization (see work on the “Hadamard parameterization” in implicit sparse regularization), allowing the network to select relevant wavelet features.

[+] The paper is well-organized and cleanly written. The authors revision has done a good job of addressing all clarity concerns of reviewers. 

[-] Several reviewers raised concerns about the limited scope of the experiments: the paper only replaces a single layer of a particular architecture (AlexNet) and evaluates on one particular dataset (ImageNet). 

[-] The main proposed benefit of this modification is in the interpretability of the network and its potential amenability to mathematical analysis. This claim would be stronger if the paper either 1. showed the benefit by exhibiting some rudimentary mathematical theory for this network or 2. used this idea to demonstrate networks that are significantly more interpretable, say by replacing all learned convolutional layers with DT-CWPT. 

The paper’s reviews were split. All reviewers appreciate the paper’s clean exposition of a reasonable idea, and note the novelty of using dual tree wavelets in this context. However, reviewers express concerns about the paper’s significance: it could do more to show how replacing the lowest layer with DT-CWPT yields new insights, and do more to demonstrate (both rhetorically and experimentally) the generality of its ideas. Based on the bulk of the reviews, as written the paper falls slightly below the threshold for acceptance. 
Decision: Reject