A NEW POINTWISE CONVOLUTION IN DEEP NEURAL NETWORKS THROUGH EXTREMELY FAST AND NON PARAMETRIC TRANSFORMS
Authors: Joonhyun Jeong
Sung-Ho Bae
Venue:  (2020)
Keywords: Pointwise Convolution
Discrete Walsh-Hadamard Transform
Discrete Cosine-Transform

Abstract:     Some conventional transforms such as Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have been widely used as feature extractors in image processing but rarely applied in neural networks. However, we found that these conventional transforms have the ability to capture the cross-channel correlations without any learnable parameters in DNNs. This paper firstly proposes to apply conventional transforms on pointwise convolution, showing that such transforms significantly reduce the computational complexity of neural networks without accuracy performance degradation. Especially for DWHT, it requires no floating point multiplications but only additions and subtractions, which can considerably reduce computation overheads. In addition, its fast algorithm further reduces complexity of floating point addition from O(n^2) to O(nlog n). These non-parametric and low computational properties construct extremely efficient networks in the number parameters and operations, enjoying accuracy gain. Our proposed DWHT-based model gained 1.49% accuracy increase with 79.4% reduced parameters and 48.4% reduced FLOPs compared with its baseline model (MoblieNet-V1) on the CIFAR 100 dataset.

Under review as a conference paper at ICLR 2020
A NEW POINTWISE CONVOLUTION IN DEEP NEURAL NETWORKS THROUGH EXTREMELY FAST AND NON PARAMETRIC TRANSFORMS
Anonymous authors Paper under double-bling review
ABSTRACT
Some conventional transforms such as Discrete (DWHT) and Discrete Cosine Transform (DCT) have been widely used as fea- ture extractors in image processing but rarely applied in neural networks. How- ever, we found that these conventional transforms have the ability to capture the cross-channel correlations without any learnable parameters in DNNs. This pa- per firstly proposes to apply conventional transforms on pointwise convolution, showing that such transforms significantly reduce the computational complexity of neural networks without accuracy degradation. Especially for DWHT, it re- quires no floating point multiplications but only additions and subtractions, which can considerably reduce computation overheads. In addition, its fast algorithm further reduces complexity of floating point addition from (n O(n log n). These non-parametric and low computational properties construct extremely ef- ficient networks in terms of the number of parameters and operations, enjoying accuracy gain. Our proposed DWHT-basch model gained 1.49% accuracy in- crease with 79, 99.4% reduced parameters and 49 49.4% reduced FLOPs compared with its baseline model (MoblieNet-V1) on the CIFAR 100 dataset.
Transform
INTRODUCTION
Krizhevsky et al.) 2012, Simonyan & Zisserman 2014 He et al.] 2016; Szegedy et al., 2016b and automatic Neural Architecture Search (NAS) based networks (Zoph et al. 2011 8: Liu et al. 2018 Real et al. 2018 have evolved to show remark- able accuracy various tasks such image classification (Deng al. 2009 Krizhevsky & Hinton 2009), object detection (Lin et al. 2014), benefited from huge amount of learnable parameters and computations. However, these large number of weights and high computational cost enabled only limited applications for mobile devices that require the constraint on memory space being low as well as for devices that require real-time computations (Canziani et al. 2016).
Large Convolutional Neural Networks (CNNs)
With regard to solving these problems, Howard et al. 2017): Sandler et al. (2018) Zhang et al. (2017b); Ma et al. (2018) proposed parameter and computation efficient blocks while maintaining almost same accuracy compared to other heavy CNN models. All of these blocks utilized depthwise separable convolution, which deconstructed the standard convolution with the (3
C' size for each kernel into spatial information specific depthwise convolution (3 x 3 x 1) and channel infor- mation specific pointwise (1 x x C) convolution. The depthwise separable convolution achieved comparable accuracy compared to standard spatial convolution with hugely reduced parameters and FLOPs. These reduced resource requirements made the depthwise separable convolution as well as pointwise convolution (PC) more widely used in modern CNN architectures.
Nevertheless, we point out that the existing PC layer is still computationally expensive and occupies lot of proportion in the number of weight parameters (Howard al. 2017). Although the demand toward PC layer has been and will be growing exponentially in modern neural network architectures, there has been little research improving the naive structure of itself.
Therefore, this paper proposes a new PC layer formulated by non-parametric and extremely fast conventional transforms. Conventional transforms that we applied CNN models Discrete

---

Under review as a conference paper at ICLR 2020
Walsh- Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT), which have widely been used in image processing but rarely been applied in CNNs (Ghosh & Chellappa 2016).
We empirically found that although both of these transforms do not require any learnable parameters at all, they show the sufficient ability to capture the cross-channel correlations. This non-paramen -onaamamcttric property enables our proposed CNN models to be significantly compressed in terms of the number of parameters, leading to get the advantages (i.e. efficient distributed training, less communication between server and clients) referred by [andola al.] (2016). We note that especially DWHT is considered to be a good replacement of the conventional PC layer, as it requires no floating point multiplications but only additions and subtractions by which the computation overheads of PC lay- ers can significantly be reduced. Furthermore, DWHT can take a strong ad of its fast ver- sion where the computation complexity of the floating point operations is reduced from O(n²) to On n). These non-parametric and low computational properties construct extremely efficient neural network from the perspective of parameter and computation as well as enjoying accuracy gain.
Our contributions are summarized as follows:
We propose a new PC layer formulated with conventional transforms which do not require any learnable parameters well as significantly reducing the number of floating point operations compared to the existing PC layer.
The great benefits of using the bases of existing transforms come from their fast versions, which drastically decrease computation complexity in neural networks without degrading accuracy.
We found that applying ReLU after conventional transforms discards important information extracted, leading to significant drop in accuracy. Based on this finding, we propose the optimal computation block for conventional transforms.
We also found that the conventional transforms can effectively be used especially for ex- tracting high-level features in neural networks. Based on this, we propose a new transform- based neural network architecture. Specifically, using DWHT, our proposed method yields 1.49% accuracy gain as well as 79.4% and 49.4% reduced parameters and FLOPs, respec- tively, compared with its baseline model (MobileNet-VI] on CIFAR 100 dataset.
2 RELATED WORK
2.1
DECONSTRUCTION AND DECOMPOSITION OF CONVOLUTIONS
For reducing computation complexity of the existing convolution methods, several approaches of rethinking and deconstructing the naive convolution structures have been proposed. [Simonyan & Zisserman 2014) factorized large sized kernel (c.g. 5 x 5) in convolution layer into several small size kernels (e.g. 3 x 3) with several convolution layers. Jeon & Kim (2017) pointed out the limitation of existing convolution that it has the fixed receptive field. Consequently, they introduced learnalle spatial displacement parameters, showing flexibility of convolution. Based on onJeenn & Kim (2017). Jeon & Kim (2018) proved that the standard convolution can be deconstructed as a single PC layer with the spatially shifted channels. Based on that, they proposed a very efficient convolution layer, namely active shift layer, by replacing spatial convolutions with shift operations.
It is worth noting that the existing PC layer takes the huge proportion of computation and the num- ber of weight parameters in modern lightweight CNN models (Howard et al. 2017; Sandler al. 2018 Ma et al. (018)). Specifically, MobileNet-VI (Howard et al. 2017) requires 94%, 74% of the overall computational cost and the overall number of weight parameters for the existing PC layer, respectively. Therefore, there were attempts to reduce computation complexity of PC layer. iZhang] ctt al. (001bb) proposed ShuffleNet-V where the features are decomposed into several groups over channels and PC operation was conducted for each group, thus reducing the number of weight pa- rameters and FLOPs by the number of groups G. However, it was proved in Ma et al. (2018) that the memory access cost increases G increases. leading slower inference speed. Similarly the aforementioned methods, our work is to reduce computation complexity and the number of weight parameters in convolution layer. However, our objective is more oriented finding mathe-
2

---

Under review as a conference paper at ICLR 2020
matically efficient algorithms that make the weights in convolution kernels more effective in feature representation as well as efficient in terms of computation.
2.2 QUANTIZATION
Quantization in neural networks reduced the number bits utilized to represent the weights and/or activations, Vanhoucke et al. (201T) applied 8-bit quantization weight parameters, which enabled considerable speed-up with small drop of accuracy. Gupta et al. (2015) applied 16-bit fixed point representation with stochastic rounding. Based on Han et al. (201 5b) which pruned the unimpor- lant weight connections through thresholding the values of weight, Han et al. (2015a) successfully combined the pruning with bits or less quantization and huffman encoding. The extreme case of quantized networks was evolved from Courbariaux et al. (2015), which approximated weights with the binary (+1,- 1) values. From the milestone of Courbariaux et al. (2015), (11s).(ourbariuu & Bengio 2016); |ubara al. (2016) constructed Binarized Neural Networks which either stochastically or deterministically binarize the real value weights and activations. These binarized weights and acti- vations lead to significantly reduced run-time by replacing floating point multiplications with 1-bit XNOR operations.
Based on Binarized Neural Networks (Courbariaux & Bengio, 2016 Hubara al. 2016), Local Binary CNN (Juefci-Xu et al., 2016) proposed a convolution module that utilizes binarized non- learnable weights in spatial convolution based Local Binary Patterns, thus replacing multiplica- tions with addition/subbraacionn operations in spatial convolution. However, they did not consider reducing computation complexity in PC layer and remained the weights of PC layer learnable float- ing point variables. Our work shares the similarity to Local Binary CNN (Juefei-Xu et al. 2016) in using binary fixed weight values. However, Local Binary Patterns have some limitations being applied in CNN since they can only be used in spatial convolution as well as there are approaches that enable fast computation of them.
2.3 CONVENTIONAL TRANSFORMS
In general, several transform techniques have been applied for image processing. Discrete Cosine Transform (DCT) has been used as powerful feature extractor (Dabbaghchian et al.] 2010). For N N-point input sequence, the basis kernel of DCT defined as a list of cosine values as below:
(2x + 1)mm
Cn
|cos(
DSISN-1
2N
where is the index of a basis and captures higher frequency information in the input signal as m increases. This property led DCT to be widely applied in image/video compression techniques that emphasize the powers of image signals in low frequency regions (Rao Yip 2014).
Discrete Walsh Hadamard Transform (DWHT) is very fast and efficient transform by using only +1 and -1 elements in kernels. These binary elements kernels allow DWHT to perform with- out any multiplication operations but additiinn/subrratiion operations. Therefore, DWHT has been widely used for fast feature extraction in many practical applications, such as texture image segmen- tation (Vard et al. 2.1I), face recognition (Hassan et al. 2007), and video shot boundary detection
G. & S. 2014).
Further, DWHT can take advantage of structured- wwrbeeeddwiiin--bbe wiring-based fast algorithm (Algorithm as well as allowing very high efficiency in encoding the spatial information (Pratt et al. 1969) The basis kernel matrix of DWHT is defined using the previous kernel matrix below:
P HD = DD - -
where H 1 and D > 1. In this paper denote HD as the m-th vector of H in Eq- 2 Additionally, we adopt fast DWHT algorithm to reduce computation complexity of PC layer in neural networks, resulting in extremely fast and efficient neural network.
3

---

Under review as a conference paper at ICLR 2020
3 METHOD
We propose a new PC layer which computed with conventional transforms. The conventional PC layer can be formulated as follows:
Ziym = WT X
1 m S M
(3)
where (i. j) is a spatial index, and m output channel index. In Eq. 3 N and M are the number of input and output channels, respectively. Xiy E R is a vector of input X at the spatial index (i, j), Wm. E R. is vector of m-th weight W in Eq. For simplicity, the stride is set as 1 and the bias omitted in Eq.
Our proposed method is to replace the learnable parameters Wmn with the bases in the conven- tional transforms. For example, replacing W., with H in Eq. we now can formulate the new multiplication-free PC layer using DWHT. Similarly, the DCT basis kernels Cn in Eq. [ can sub- formulating another new PC layer using DCT. Note that the normalization factors in the conventional transforms are not applied in the proposed PC layer, because Batch Nor- malization (Ioff Szegedy 2013) performs a normalization and a linear transform which can be viewed as normalization in the existing transforms.
stitute for Wm. m in Eq. 3
The most important benefit of the proposed method comes from the fact that the fast algorithms of the existing transforms can be applied in the proposed PC layers for further reduction of computa- tion. Directly applying above new PC layer gives computational complexity of O Adopting the fast algorithms, can significantly reduce the computation complexity of PC layer from O (N2) to O(NoggN) without any change of the computation results.
We demonstrate the pseudo-code of our proposed fast PC layer using DWHT in Algorithm D based on the Fast DWHT structure shown in Figure Ta In Algorithm [ for log. iterations, the even- indexed channels and odd-indexed channels spectively. The resulting elements which were added and subtracted are placed in the first N/2 elements and the last N/2 elements of the input of next iteration, respectively. this computation process, each iteration requires only N operations of additions or subtractions. Consequently, Al- gorithm D gives us complexity of ONNoggN) in addition or subtraction. Compared to the existing PC layer that requires complexity of (N2) in multiplication, our method is extremely cheaper than the conventional PC layer in terms of computation costs as shown Figure Tbb and in power consumption of computing devices (Horowitz) 2014). Note that, similarly to fast DWHT, DCT can also be computed in a fast manner that recursively decomposes the N N-point input sequence into two subproblems of N/2-point DCT (Kok 1997).
added and subtracted in element-wise manner, re-
Compared to DWHT, DCT takes advantage of using more natural shapes of cosine basis kernels, which tend to provide better feature extraction performance through capturing the frequency infor- mation. However, DCT inevitably needs multiplications for inner product between C and X vectors, and look up table (LUT) for computing cosine kernel bases which increase the processing time and memory access. On the other hand, as mentioned, the kernels of DWHT consist only of +1, -1 which allows for building multiplication-ffre module. Furthermore, any memory access towards kernel bases is not needed if our sruutturedwwirnng-beeee applied. Our comprehensive experiments in Section 3.1 and 3.2 show that DWHT' more efficient than DCT in being applied in PC layer in terms of trade-off between the complexity of computation cost and accuracy.
fast DWHT algorithm (Algorithm M is
Note that, for securing more general formulation of our newly defined PC layer, we padded zeros along the channel axis if the number of input channels is less than that of output channels while truncating the output channels when the number of output channels shrink compared to that of input channels as shown in Algorithm
Figure Ia shows the architecture of fast DWHT algorithm described in Algorithm| I This structured- wiring- based architecture ensures that the receptive field of each output chamnels is N, which means each output channel is fully reflected against all input channels through log2N iterations. This fully- reflected property helps to capture the input channel correlations in spite of the computation process of what channel elements will be added and subtracted being structured in deterministic manner.
4

---

Under review as a conference paper at ICLR 2020
For successfully fusing new PC layer into neural networks, explored two themes: an opti- mal block search for the proposed PC; ii) an optimal insertion strategy of the proposed block found by i), in hicrarchical manner the blocks of networks. We assumed that there block unit structure and an optimal hierarchy level (high-, middle-, low-level) blocks in the neural networks favored by these non-learnable transforms. Therefore, conducted the experiments for the two affremennioned themes accordingly. We evaluated the effectivvnness for each of our networks by accuracy fluctuation the number of learnable weight parameters or FLOPs changes. For com- parison, we counted total FLOPs with summation of the number of multiplications, additions and subtractions performed during the inference. Unless mentioned, we followed the default experimen- tal setting as 128 batch size, 200 training epochs, 0.1 initial learning rate where 0.94 is multiplied per epochs, and 0.9 momentum with 5e-4 weight decay value. In the experiments, the model ac- curacy was obtained by taking average of Top-1 accuracy values from three independent training results.
optimal
Conventional PC
PC based on fast DWHT
muttipliation operations
20
PC based on fast DCT
8
10
(a) black circle indicates channel element, and black and red lines are additions and subtractions, respectively. The number of input channels is set for simplicity. Best viewed in color.
Number input channels: 2" n
(b) æ axis denotes logarithm of the number of in- put channels which range from 2 to 2". For sim- plicity, the number of output channels is set to be same that of the input channel for all PC layers. viewed color.
Figure 1: Left: architecture of our PC layer based fast DHWT algorithm in Algorithm 1 Right: comparison of the number of multiplications between our new PC layers and the conventional PC layer.
Algorithm A new pointwise convolution using fast DWHT algorithm
Input: 4D input features X(B N H W), output channel M
1: n logg N
2: if N then
3:
ZeroPadID(X,, axis=l)
b> pad zeros along the channel axis
4: end if
5: for +-
do
6:
< :f:::;2;;1.
7:
o X|:, 1: 2,;]
8:
+ X/.N/2 ; 0
X|:,:
9:
10:
end for
11:
if N> M then
12:
X X [ M,::]
13: end if
5

---

Under review as a conference paper at ICLR 2020
3.1 OPTIMAL BLOCK STRUCTURE FOR THE CONVENTIONAL TRANSFORMS
From a microscopic perspective, the block unit is the basic foundation of neural networks, and it determines the efficiency of the weight parameter space and computation costs in terms of accuracy. Accordingly, to find the optimal block structure for our proposed PC layer, our experiments are con- ducted to replace the existing PC layer blocks with our new PC layer blocks in ShuffleNet- V2 (Mal et al. 2018). The proposed block and its variant blocks are listed in Figure 2 Comparing the results of (c) and (d) in Table [ informs us the important fact that the ReLU (Nair & Hinton 2010) activa- tion function significantly harms the accuracy of our neural networks equipped with the conventional transforms. We empirically analyzed this phenomenon in Section 4.1 Additionally, comparing the
accuracy results of (b) and (d) in dedooess that the proposed PC layers superior to the PC layer which randomly initialized and fixed its weights to be non-learnable. These results imply that DWHT and DCT kernels can better extract meaningful information of cross-channel correlations compared to the kernels which are randomly initialized and non-learnable Compared to the base- line model in Table [ (d)-DCT w/o ReLU and (d)-DWHT w/o ReLU blocks show accuracy drop by approximately 2.3% under the condition that 42% and 49.5% of learnable weight parameters and FLOPs are reduced, respectively. These results imply that the proposed blocks (c) and (d) are still inefficient in trade-off between accuracy and computation costs of neural networks, leading to more explore to find out an optimal neural network architecture. In the next subsection, we address this problem through applying conventional transforms on the optimal hierarchy level features (See Section 3.2). Based on our comprehensive experiments, we set the block structure (d) as our default proposed block which will be exploited in all the following experiments.
Channel Spit
Channel Sprit
Channel Spitt
Chonnel Spit
ROPC
BN
3a5 mwtoo
3x3 DWCOww
SeB DWVCom
3a5 DWCon
Tut Cotts
RCPC
CTPC
CTPC
ReUU
Concat
Concat
Concal
Cencal
Chamnel Shuffe
Channel Shaffe
Chanrel Stufte
ChamelStuffe
(a)
(b)
(c)
Figure 2: Structure of block units. (a): the basic block of ShuffleNet- V2: (b): block using random constant pointwise convolution (RCPC) layers; (c): block using conventional transform pointwise convolution (CTPC) layers with ReLU applied after each of CTPC layer; (d): our proposed block using CTPC layers without ReL... Block (b) randomly initialized the weights of PC layer with the distribution of U(-1/VN/2, /N/2), where N is the number of input channel and fixed these weights during training.
3.2 OPTIMAL HIERARCHY LEVEL BLOCKS FOR CONVENTIONAL TRANSFORMS
In this section, we search on an optimal hierarchy level where our optimal block which is based on the proposed PC layer is effectively applied in whole network architecture. The optimal hierarchy level will allow the proposed network to have the minimal number of learnable weight parameters and FLOPs without accuracy drop, which is made possible by non-parametric and extremely fast conventional transforms. It is noted that applying our proposed block on the high-level blocks in the network provides much more reduced number of parameters and FLOPs rather than applying on low-level blocks, because channel depth increases exponentially as the layer goes deeper in the network.

---

Under review as a conference paper at ICLR 2020
Model
Top-1 Ace (%)
Weights (ratio)
of FLOPS (ratio)
(a)-baseline
71.68 + 0.26
1.57M (Ix)
102.9M (Ix)
(b)-RCPC w/o ReLU
68.16 1 0.07
0.92M (0,58x)
102.9M (Ix)
(c)-DWHT ReLU
66.2 0.22
0.92M (0.58x)
50.2M (0.48x)
(c)-DCT w/ ReLU
66.55 + 0.5
0.92M (0.58x)
54.7M (0.53x)
(d)-DWHT w/o ReLU 69.42 + 0.31
0.92M (0,58x)
50.2M (0,48x)
(d)-DCT w/o ReLU
69.23 + 0.14
0.92M (0.58x)
54.7M (0.53x)
Table I: Performance result of block units in Figure on CIFARI00 dataset. All the experimented models are based on ShuffleNet-V2 with width hyper- parameter 1.1x which we customized to make number of output channels in Stage2, 3, as 128, 256, 512, respectively for fair comparison with DWHT which requires 2 input channels. We replaced all of 13 stride 1 basic blocks (i.e. (a) block) in baseline model with (b), (c), (d) blocks, respectively. (c)-DWHT w/ ReLU denotes CTPC layer in (c) block is based on DWHT, while (d)-DCT w/o ReLU denotes CTPC layer in (d) block is based on DCT.
73
72
72-
71
Accunicy
70
70
70-
69
Top.1
68
68
68
Secre CMI
Ruddine
Roadine
DWVIT www.t/lllllae..
DGMI
0.5
1.0
3.5
2.0
2.5
0.5
1.0
1.5
2.0
2.5
0.5
1.0
1.5
2.0
2.5
Params (M)
Farams (M4)
Farams (M)
73
73
73
100- Accuracs
71
71
73
60
69
69
Rucchn canil or High loprs
Brachee
67
67
DCT
DWKIT do levll wvallayerr
D0NF on low hoalliyyes
100
150
100
150
50
100
150
FLOPs
Total
Figure 3: Performance curve of hierarchically applying our optimal block on CIFARI00, Top: in the viewpoint of the number of learnable weight parameters, Bottom: in the viewpoint of the number of FLOPs. The performance of baseline models was evaluated by ShuffleNet- V2 architecture with width hyper-parameter 0.5x, 1x, 1.1x, 1.5x. Our models were all experimented with 1.1x setting, and each dot the figures represents mean accuracy 3 network instances. Note that the blue line denotes the indicator of the efficiency of weight parameters FLOPs in terms of accuracy. The upper left part from the blue line is the superior region while lower right part from blue line is the inferior region compared to the baseline models.
In Figure 3 we applied our optimal block (i.e. (d) block in Figure 2 on high-. middle- and low-level blocks, respectively. In our experiments, we evaluate the performance of the networks depending on the number of blocks where the proposed optimal block is applied. The model that we have tested denoted as (transform type)-(# of the proposed blocks)- level in Low (I.), Middle (M), and High (H) where the proposed optimal block is applied), For example, DWHT-3-L. indicates the neural network model where the first three blocks ShuffleNet-V V2 consist the proposed blocks, while the other blocks are the original blocks of ShuffleNet-V3 It is noted that in this experiment, we fix all the blocks with stride 2 in the baseline model to be original ShuffleNet-V V2 (Ma et al. 2018) stride 2 blocks.
Figure shows the performance of the proposed methods depending on the transform types (DCT, DWHT), hierarchy levels (I., M, H} and the number of the proposed blocks that replace the original

---

Under review as a conference paper at ICLR 2020
ones in the baseline (3, 6, 10) in terms of Top-1 accuracy and the number of learnable weight parameters (or FLOPs). It is noted that, since the baseline model has only 7 blocks in the middle- level Stage (i.e. Stage3), performed the middle-level experiments only for DCT/DWHT-3-M and -7-M models where the proposed blocks are applied from the end of Stage3 in the baseline model. In Figure the performance of our 10-H (or 10-L.), 6-H (or 6-I.), 3-H (or 3-L.) models (7-M and 3-M only for middle-level experiments) is listed in ascending order of the number of learnable weight parameters and FLOPs.
As shown in the first column of Figure 3 applying our optimal block on the high-level blocks achieved much better trade-off between the number of learnable weight parameters (FLOPs) and accuracy. Meanwhile, applying on middle- and low-level features suffered. respectively, slightly and severely from the inefficiency of the number of weight parameters (FI.Oss) with regard to accuracy. This tendency is shown similarly for both DWHTT-based models and DCT-based models, which implies that there can be an optimal hierarchical level of blocks favored by conventional transforms. Also note that our DWHT-based models showed slightly higher or same accuracy with less FLOPs in all the hierarchy level cases compared to our DCT-based models. This is because the fast version of DWHT does not require any multiplication but needs less amount of addition or subtraction operations compared to the fast version of DCT while it also has the sufficient ability to extract cross-channel information with the exquisite wiring-based structure.
For verifying the generality of the proposed method, we also applied our methods into MobileNet- V1 (Howard et al. 2017). Inspired by the above results showing that optimal hierarchy blocks for conventional transforms can be found in the high-level blocks, we replaced high-level blocks of baseline model (MobileNet-VII and changed the number of proposed blocks that are replaced to verify the effectiveness of the proposed method. The experimental results are described in Table 2 Remarkably, as shown in Table 2 our DWHT-6-H model yielded the 1.49% increase in Top-1 accuracy even under the condition that the 79.4% of parameters and 49.4% of FLOPs are reduced compared with the baseline model. This outstanding performance improvement comes from the depthwise separable convolutions used in MobileNet-V1, where PC layers play dominant roles in computation costs and memory space, i.c. they consume 94.86 94.86% in FLOPs and 74% in the total number of parameters in the whole network (Howard et al. 2017). The full performance results for all the hierarchy levels (L, M. H} and the number of blocks 33 6, 10) (exceptionally, (3. 7} blocks for the middle level experiments) are described in Appendix A
In Appendix A based on the comprehensive experiments it can be concluded that i) the proposed PC block always shows its better efficiency of number of parameters and FLOPSs when applied on high-levels compared to when applied on low-level in the network hierarchy; ii) the performance gain start to decrease when the number of transform based PC blocks exceeded certain capacity of the networks.
Model
Top-1 Acc (%)
# of Weights (ratio)
# of FLOPs (ratio)
Baseline
67.15 1 0.3
3.3IM (Ix)
92.4M (Ix)
DWHT-3-H
68.19 0.35
1.47M (0.44x)
71.6M (0.77x)
DCT-3-H
68.21 = 0.19
1.47M (0.44x)
72M (0.78x)
DWHT-6-H
68.65 + 0.27
0.68M (0.2x)
46.7M (0.5x)
DCT-6-H
67.95 + 0.53
0.68M (0.2x)
47.7M (0.5Ix)
Table 2: Performance result of hierarchically applying our optimal block on CIFAR100 dataset. All
the models are based on MobileNet-Vi with width hyper- parameter 1x. We replaced both stride 1, 2 blocks in the baseline model with the optimal block that consist of [3 x 3 depthwise convolution Batch Normalization ReLU CTPC- Batch Normalization] in series.
4 EXPERIMENTS AND ANALYSIS
In this section, analyze the significant accuracy degradation of applying ReLU after our proposed PC layer. Additionally, we analyze the active utilization of 3x3 depthwise convolution weight kernel values which takes an auxiliary role for conventional transform being non-leamable.
8

---

Under review as a conference paper at ICLR 2020
High level
High level Middle level
8%
Middle level
10%
Low level
Low level
6%
8%
4%
5%
0%
0%
-15 -10
-5
10
15
20
-2 -1
Figure 4: Histograms hierarchy level (low-level, middle-level, high-level) activations after the proposed PC layer based on conventional transforms, Left: DWHT, Right: DCT. Both DWHT and DCT models are based on ShuffleNet V2 1.1x model where we replaced all of stride blocks with (d)-DWHT w/o ReLU and (d)-DCT w/o ReLU blocks, respectively in Figure 2
25%
Baseline
DCI13-H
20%
DWHHT-3-
15%
5%
0%
-0.10
-0.05
0.00
0.05
0.10
0.15
Figure 5: Histogram of depthwise convolution weights in the third block, out of last blocks. DCT-3-H and DWHT-3-H models are based on ShuffleNet V2 1.1x model with (d) block. Baseline model ShuffleNet V2 1.1x model.
4.1 HINDRANCE OF RELU IN CROSS-CHANNEL REPRESENTABILITY
As shown in Table D applying ReLU after conventional transforms significantly harmed the accu- racy. This is due to the properties of conventional transform basis kernels that both H in Eq. and C. in Eq. Thave the same number of positive and negative parameters in the kernels except for m and that the distributions of absolute values of positive and negative elements in kernels are almost identical. These properties let us know that the output channel elements that have under zero value should also be considered during the forward pass; when forwarding X., in Eq. through the conventional transforms if some important channel elements in Xij that have larger values than others are combined with negative values of Cm H the important cross-channel information the output Zijm in Eq. can reside the value range under zero. Figure shows that all the hierarchy level activations from both DCT and DWHT based PC layer have not only positive values but also negative values in almost same proportion. These negative values possibly include impor- tant cross-channel correlation information. Thus, applying ReLU on activations of PC layers which are based on conventional transforms discards crucial cross-channel information contained neg- ative values that must be forwarded through, leading to significant accuracy drop shown in the results of Table [ Figure @empiricll demonstrates above theoretical analysis by showing that as the negative value regions are fully ignored (i.e. ReLU), the accuracy is significantly degraded while fully reflecting the negative value regions (i.e. g 1) shows the best accuracy. From above kernel value based analysis and its experiment, we do use non-linear activation function after the proposed PC layer.
4.2 ACTIVE x 3 DEPTHWISE CONVOLUTION WEIGHTS
In Figure 5 and Appendix it is observed that 3 3 depthwise convolution weights of last 3 blocks in DWHT-3-H and DCT-3-H have much less near values than that of baseline model. That

---

Under review as a conference paper at ICLR 2020
100- Mecurer (14
0.0 0.2
0.4
08 1.0
2 Inegative
Figure 6: Ablation study of negative slope term in activation function F, which is defined as F(x) mar(0, x) g min æ). The performance of models were evaluated based on DCT or DWHT-13-H ShuffleNet- V2 1.1x where we applied as an activation function after every DCT or DWHT based PC layer and Batch Normalization layer.
DCT-3H
77.25
DWVT-3-1
72.00
73.75
21.00
11.45
70.15
0.00
2.22
1.44
0,00
0.05
0.30
Figure 7: Ablation study of weight decay values (5e-4, 2e-3, 1e-2, le-1). We applied these weight decay values only on 3 3 depthwise convolution weights of last 3 blocks in DCT-based model and DWHT-based model, while all the other learnable weights were regularized with weight decay of 5e-4.
is, the number of values which are apart from near-zero is much larger on DCT-3-H and DWHT- 3-H models than on baseline model. We conjecture that these learnable weights whose values apart from near-zero were actively fitted to the optimal domain that is favored by conventional transforms. Consequently, these weights actively and sufficiently utilized to take the auxiliary role for conventional transforms which are non-learnable, deriving accuracy increase compared to the conventional PC layer shown in Figure 3
To verify the impact of activeness of these 3 x depthwise convolution weights in the last blocks, experimented with regularizing these weights varying the weight decay values. Higher weight decay values strongly regularize the scale of 3 x 3 depthwise convolution weight values in the last blocks. Thus, strong constraint on the scale of these weight values hinders active utilization of these weights, which results in accuracy drop as shown in Figure 7
5 CONCLUSION
We propose the new PC layers through conventional transforms. Our new PC layers allow the neural networks to be efficient in complexity of computation and learnable weight parameters. Especially for DWHT-based PC layer, its floating point multiplication-fre property enabled extremely effi- cient in computation overhead. With the purpose of successfully fusing our PC layers into neural networks, we empirically found the optimal block unit structure and hierarchy level blocks in neu- ral networks for conventional transforms, showing accuracy increase and great representability in cross-channed correlations. We further intrinsically revealed the hindrance of ReLU toward captur- ing the cross-chame cross-channel representability and the activeness of depthwise convolution weights on the last blocks in proposed neural network.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. An analysis of deep neural network models for practical applications. CoRR, abs/1605.07677. 2016. URL httphttt////xiv.oet
abs/ 07678
Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830, 2016. URL http::/aaax..i
org/aba/1602. 02830
Matthieu Courbariaux, Yoshua Bengio, and networks with binary weights during propagations. CoRR, abs/1511.00363 2015. URL http:
David. Binaryconnect: Training deep neural
// /aaxiivorggabbs///// http://aaaa111..0110 00363
Saeed Dabbaghchian, Masoumeh P Ghaemmaghami, and Ali Aghagolzadeh. Feature extraction using discrete cosine transform and discrimination power analysis with face recognition tech- nology. Pattern Recognition, 43(4):1431 1431-1400,, 2010.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Icee, 2009.
L. G. G. and D. S. Walshhadamard transform kernel-based feature vector for shot boundary detection. IEEE Transactions Image Processing, 23(12):5187-5197, Dec 2014. ISSN 1057- 7149. doi: 10. 2..10999T122211422666562.
Arthita Ghosh and Rama Chellappa. Deep feature extraction in the det domain. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 3536-3541. IEEE, 2016.
Suyog Gupta, Ankur Agrawal. Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 15 10.00/49, 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances neural information processing systems, pp. 1135-1143, 1443, 2015b.
M Hassan, I Osman, and M Yahia. Walsh-hadamard transform for facial feature extraction in face recognition. World Academy of Science, Engineering and Technology, 1994 2007.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016. doi: 10. 10.1109/CVPR.2016.0
M. Horowitz. 1.1 computing's 's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14, Feb 2014. doi: 10. 00.1109//SSCC2224.667333333
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861,, 2017. URI.
1704. 04861
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran Yl-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. CoRR, abs/1609.07061 2016. URL http: /ttp/vaiiiv/..////.//eae//aaatla org/ abs/ 11w.. 07061
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint /602 2016.
11

---

Under review as a conference paper at ICLR 2020
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. URL (att:://aaxxxii/// abs/1502. 03167
Vidit Jain and Erik Leamed- Fddb: A benchmark for face detection in unconstrained settings. 2010.
Yunho Jeon and Junmo Kim. Active convolution: Learning the shape of convolution for image clas- sification. CoRR, abs/1703.0900 2017.
09076
Yunho Jeon and Junmo Kim. Constructing fast network through deconstruction of convolution. CoRR, abs/1806.07370, 2018.
Felix Juefei-Xu, Vishnu Naresh Boddeti, and Marios Savvides, Local binary convolutional neural networks. CoRR, 0sss/6608.000499 2016.
Chi-Wah Kok. Fast algorithm for computing discrete cosine transform. IEEE Transactions on Signal Processing, 45(3):757-760 1997.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. Proceed- ings of the European Conference Computer Vision (ECCV), pp. 19-34, 2018.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. CoRR, abs/1807. 11164, 2018. URL hatt:://aaaee
org bogg/abs/in 1807. 11164
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
W. K. Pratt, J. Kane, and H. C. Andrews. Hadamard transform image coding. Proceedings of the IEEE, 57(1):58-66 68, Jan 1969. ISSN 0018-9219, doi: 10.1 19W.PRO..1999.688800
K Ramamohan Rao and Ping Yip. Discrete cosine transform: algorithms, advantages, applications. Academic press, 2014.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. arXiv preprint 1rXiv:1802.015433 2018.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and seg- mentation. CoRR, 0bb//8801.04/ 104381, 2018. URL/attp://axxxi wttp:/iiiv..g//a////beaae 04381]
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arxiv::409 1556, 2014.
Christian Szegedy, Sergey loffe, and Vincent Vanhoucke. Inception-v4, inception-resnel and the impact of residual connections on learning. CoRR, N0W16002.0721.. 2016a. URL http://
Trxiv.org/l httgg/bbs/// http:/ab/s/11.0..000
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2818-2826, 2016b.
12

---

Under review as a conference paper at ICLR 2020
Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks cpus. 2011.
AliReza Vard, AmirHassan Monadjemi, Kamal Jamshidi, and Naser Movahhedinia. Fast texture energy based image segmentation using directional walsh-hadamard transform and parametric actíve contour models. Expert Systems with Applications, 38(9):11222-1177)) 2011.
Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. Wider face: A face detection bench- mark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5525-5533, 2016.
Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Zi. S3fd: Single shot scale-invariant face detector. In Proceedings of the IEEE International Conference on Computer Vision, pp. 192-201, 2017a.
Xiangyu Zhang. Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. CoRR, abs/1707.01083, 2017b. URL http:
//rxxiv.orrg/ass//777.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8697-8710, 2018.
13

---

Under review as a conference paper at ICLR 2020
A GENERALITY OF APPLYING PROPOSED PC LAYERS IN OTHER NEURAL NETWORKS
67.5
67.5
57.5
65.0
65.0
55.0
60.0
02.0
00.0
Inselit
Sanel ta
35.0
DOBIT sgriiv
DOETT animts
70.0
70.0
70.0
05.0
03.0
05.0
$0.0
62.0
57.5
57.5
57.5
25
75
300
125
150
75
100
125
150
50
75 Total FLOPs I80
300
125
150
Total FLOPs (K)
Total FLOPS (14)
Figure 8: Performance curve of hierarchically applying our optimal block (See Table 2 for detail settings) CIFAR100, Top: in the viewpoint of the number of learnable weight parameters, Bot- tom: in the viewpoint the number FLOPs. The performance of baseline models was evaluated by MobileNet-Vi architecture with width hyper-parameter 0.2x, 0.35x, 0.5x, 0.75x, 1x, 1.25x. Our proposed models were all experimented with Ix setting, and each dot in the figures represents mean accuracy of 3 network instances. Our models experimented with 10-H, 6-H, 3-H models (first column). 7-M, 3-M-Rear, 3-M-Front models (second column) and 10-L, 6-L, 3-L models (final column), listed in ascending order of the number of learnable weight parameters and FLOPs.
In Figure 8 for the purpose of finding more definite hierarchy level of blocks favored by our proposed PC layers, we subdivided our middle level experiment scheme; DCT/DWHT-3-M-Fron model denotes the model which applied the proposed blocks from the beginning of Stage3 in the baseline while DCT/DWHT-3-M --aar model denotes the model which applied from the end of Stage3. The performance curves of all our proposed models in Figure [ show that if we apply the proposed optimal block within the first 6 blocks the network, the Top-1 accuracy is mildly or significantly deteriorated compared to the required computational cost and number of learnable pa- rameters, informing us the important fact that there are the definite hierarchy level blocks which are favored or not favored by our proposed PC layers in the network.
14

---

Under review as a conference paper at ICLR 2020
B HISTOGRAM OF 3 x 3 DEPTHWISE CONVOLUTION WEIGHTS IN HIGH-LEVEL BLOCKS
Baseline
20%
DCT-3-H
DWHT-3-H
15%
0%
-0.2
-0.1
0.0
0.1
0.2
0.3
Baseline
30%
DCT-3-H
25%
DWHT-3-H
20%
15%
10%
5%
-0.2 -0.1
0.0
0.1
0.2
Figure 9: Histograms x 3 depthwise convolution weights, Top: histogram of first block out of last blocks, Bottom: histogram of second block of last blocks. DWHT-3-H and DCT-3-H models are based on ShuffleNet-V2 1.1x model with (d)-DWHT w/o ReLU and (d)-DCT w/o ReLU block Figure respectively. Baseline model is ShuffleNet- V2 1.1x model.
15

---

Under review as a conference paper at ICLR 2020
C PERFORMANCE COMPARISON BETWEEN RCPC AND PROPOSED PC LAYERS
72-
72
72
Top Rocurao
71
100
69
60
T0p
69
cate an toedlom
DWNei lin
1.5
2.0
2.0
Parama (N)
Par Paraems (MS
Paramts (N)
73
73
73
72-
72
72-
To0 Accurncs
Top Accuricy
70-
70
70
69
ceria inwrtage lows
lia inottions. BCRC lnvel layers Total PLDP' (K0
67
67
FCNCcc lyyen
67
Total PLD's (8)
DdalLLONS (14)
Figure 10: Performance curve of hierarchically applying our proposed DCT/DWHT based PC layers and RCPC layer on CIFAR100 dataset. Experimental settings are the same as described in Figure Our DCT/DWHT-base models are experimented with 10-H, 6-H, 3-H models (first column), 7-M, 3-M models (second column) and 10-L, 6-L, 3-L models (final column), listed ascending order of the number of learnable weight parameters and FLOPs. Models equipped with RCPC layers experimented with 10-H, 6-H, 3-H models (first column), 7-M, 3-M models (second column) and 10-L, 6-L, 3-L models (final column), listed in ascending order of number of learnable parameters (top row) and Top-1 accuracy (bottom row).
For the purpose of demonstrating the superiority of our proposed DCT/DWHT based PC layers over RCPC layer in all the hierarchical (i.e. low/mid/high) level layers, we compared the performance trade-off in Figure TO It is noted that DCT/DWHT based PC layers almost always get higher accuracy than RCPC layer in all the hierarchical level layers. Comparing the distance between the orange green line with the red line in Figure I0 our DCT/DWHT based PC layers showed high efficiency in trade-off between accuracy and the computational costs or number of learnable parameters, compared to RCPC layer almost all the hierarchical levels.
16

---

Under review as a conference paper at ICLR 2020
D GENERALITY OF APPLYING OUR PROPOSED PC LAYERS ON OTHER TASKS
Model
Easy (mAP %)
Medium (mAP %)
Hard (mAP %)
# #f Weights (ratio)
#f FLOPS (ratio)
Baseline
89.74
86.3
64,11
2.32M (Ix)
129.4M (Ix)
DWHT-3-H
89.82
86.27
64.38
0.48M (0.2x)
108.6M (0.84x)
DCT3-
90.25
86.83
61.49
0,48M (0.2x)
109M (0.84x)
DWHT-6-H
89.98
86. 29
63.63
0.25M (0.1
I..IM (0.73x)
DCT-6-H
90.11
86.71
62.16
0.25M (0.1Tx)
96.3M (0.74x)
Quantitative comparison between the baseline model and our DCT/DWHTT-based models on WIDER FACE validation dataset.
Table 3:
Model
AP (%)
# #of Weights (ratio)
of FLOPs (ratio)
Baseline
94.41
2.32M (1x)
129.4M (1x)
DWHT-3-H
94.5
0,48M (0.2x)
108. 6M (0.84x)
DCT-3-H
93.79
0.48M (0.2x)
109M (0.84x)
DWHT-6-H
94.37
0.25M (0.1Ix)
95.1M (0. 73x)
DCT-6-H
93.67
0.25M (0.1
96.3M (0. (4.74x)
Table 4: Quantitative comparison between the baseline model and our proposed DCT/DWHT--aae models on FDDB dataset. AP means the true positive rate at 1,000 false positives and all the models were evaluated with discontinuous criterion in FDDB dataset.
In order to demonstrate the Nomain-generalle of the proposed method, we conducted comprehen- sive experiments on applying our proposed PC layers object detection, specifically the face detection task. For face detection schemes such as anchor design, data augmentation and feature- map resolution design, we followed Zhang al.) (2017a) which is one of the baseline methods in face detection field. It is noted that there is a huge demand on real-time face detection algorithms having high detection accuracy, which leads us to applying our PC layers to a lightweight face de- tection network. Therefore, instead of using VGG16 (Simonyan & Zisserman 2014) as backbone network as in Zhang et al. (2017)), we set MobileNet-V VI 0.25x as our baseline backbone model where extra depthwise separable blocks are added for detecting more diverse scales of face in the images. In this baseline model, we replaced the conventional PC layers within last 3, blocks with our DCT/DWHT based PC layers. We trained all the models on the WIDER FACE (Yang et al.) 2016) train dataset and evaluated on WIDER FACE validation dataset and Face Detection Data Set and Benchmark (FDDB) dataset @Jain & Learned-Miller 2010). WIDER FACE validation set has Easy, Medium and Hard subsets, which correspond to large, medium and small scale faces, respec- tively. Validation results of the baseline model and our proposed DCT/DWHT models on WIDER FACE are described in Table 3
we that, overall, our DWHT-3-H and DWHTT6-H models showed comparable or even higher mAP values than the baseline model on all the subsets (Easy, Medium, and Hard) with significantly reduced number of learnable parameters and FLOPs. Especially. DWHT-3-H model achieved 0. 0.27% higher mAP than the baseline model under the condition that 79% of parameters and 16% of FLOPS are reduced on Hard subset. Regarding DCT-3-H and DCT-6-H models, they showed a solid improvement of mAP on Easy and Medium subsets with significantly reduced number of parameters and FLOPs compared to the baseline model.
In Table 3
Additionally, we verified the effectiveness of the proposed method the FDDB dataset in Table We note that our DWHT-6-H and DWHT-3--H models showed comparable or even 0.09% higher AP than the baseline model with significantly reduced number of learnable parameters and FLOPs. On the other hand, our DCT-6-H and DCT-3-H models showed a small degree of degradation on AP compared to the baseline model, which is mild degradation considering the reduced amount of parameters and FLOPs.
Consequently, our comprehensive experiments on both WIDER FACE and FDDB datasets reveal the generality of our proposed method, enabling neural networks to be extremely lightweight and reduce the computational overhead.
17


Reviews:
[{"review_id": "r1gnXjEatr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new pointwise convolution layer, which is non-parametric and can be efficient thanks to the fast conventional transforms. Specifically, it could use either DCT or DHWT to do the transforming job and explores the optimal block structure to use this new kind of PC layer. Extensive experimental studies are provided to verify the new PC layer and experimental results show that the new layer could reduce the parameters and FLOPs while not loosing accuracy.\n\nOverall, this paper provides a promising PC option for the CV community and the experiments seems solid. Although the novelty is limited, which is just to combines the DCT/DHWT with NN, the experiments are sufficient and I am glad to see that this simple idea works in practice. \n\nConcerns: This paper is based on the practical validation. Is there any theory to support that using DCT/DHWT could achieve better performance. In Image Processing, DCT/DHWT could be used to compress images or videos, but what is the benefit to use them in computer vision?\n\n---post comments after rebuttal---\nThanks for the response. \n\nHowever, the rebuttal does not address my concerns. The logic behind the authors response is that since there is much work that uses DCT / DHWT-like features in computer vision tasks, we can also use them to replace the conventional PC layers. Unfortunately, I do not think that is the theory to support the idea described in this paper. Actually, the success of CNN architecture has already confirmed the advantage of conventional PC layer, it seems it is a backward step for the community. Moreover, I agree with Reviewer#4 that ImageNet should be used as benchmark to so the advantage of the paper. \n\nHence, after reading the response and the reviews from other reviewers, I support to reject the work.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1e_MoDtqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new pointwise convolution (PC) method which applies conventional transforms such as DWHT and DCT. The proposed method aims to reduce the computational complexity of CNNs without degrading the performance. Compared with the original PC layer, the DWHT/DCT-based methods do not require any learnable parameters and reduce the floating-point operations. The paper also empirically optimizes the networks by removing ReLU after the proposed PC layers and using conventional transforms for high-level features extraction. Experiments on CIFAR100 show that the DWHT-based model improves the accuracy and reduces parameters and FLOPs compared with MobileNet-V1.\n\nAlthough this paper is well organized and easy to follow, the novelty of the proposal seems limited and the performance improvement claimed by the author(s) is not very convincing due to the insufficiency of experiments. Firstly, the proposed method is just a manually designed and fixed 1*1 convolutional kernel, and its superiority over random initialization seems very limited as shown in Table 1. Also, the proposed method makes accuracy degrade when applied to low- and middle-level features. I wonder whether there is a more theoretical explanation for that. Moreover, the experiments are performed only on a small dataset CIFAR100. According to my own experience, the artificial convolutional kernels with some prior knowledge may work well on small datasets but tend to fail on larger ones. More experiments on larger-scale datasets like ImageNet are recommended to make results more convincing.\n\nTherefore, my decision leans to a rejection.\n\nSome questions:\n1. How is the performance when applying RCPC only to low-/middle-/high-level features? I suppose it should be proved that the proposed method is definitely better than random initialization.\n2. Why is only applying the proposed block to high-level layers working? How is the trade-off between parameters and accuracy different for each level of features?\n\nSpelling mistake:\nPage 6: in the second last paragraph, 'non-parameteric' should be 'non-parametric'.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bkx7z0H6KH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper proposes using non-parametric filters like Discrete Cosine Transform (DCT) and Discrete Walsh-Hadamard Transform (DWHT) which have been widely used as feature extractors in vision and image processing before deep learning became prevalent as layers especially to replace pointwise convolution (PC) layers in deep network architectures like ShuffleNet-v2 and MobileNet-V1. \n\nThe motivation is that using such layers can capture cross-channel correlations without addition of extra parameters that need to be learnt and by replacing PC layers which tend to make up the bulk of the parameters in such settings large reduction in number of weights and flops can be achieved with little drop (or even increase) in accuracy.\n\nThey show experiments on cifar100 datasets.\n\nComments:\n\n- The paper is overall easy to read although the writing and presentation can use some work. \n\n- I really enjoyed reading the paper though because it seems in the deep learning era we have a tendency to re-learn what we already know. This paper shows that combining our knowledge of image processing and compressed sensing with good function approximation leads to better and more compact representation learning. I think these aspects are being generally overlooked currently but wont be surprised to see more of these papers. \n\n- While reading section 3.2 it occurred to me that it might be interesting to consider throwing these layers in to a neural architecture search (NAS) algorithm and let it figure out the right architecture. (Just a suggestion for future work not asking for this in the rebuttal.)\n\n- Overall I am positive about the paper and have no major asks but if time and resources permit perhaps trying out the same experiments on ImageNet to see if the trend holds.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to utilize conventional frequency domain basis such as DWHT and DCT to replace the standard point-wise convolution, which can significantly reduce the computational complexity. The paper is generally well-written and easy to follow. However, the technical novelty seems limited as it is basically a simple combination of CNNs and traditional filters. Moreover, as reviewers suggested, it is our history and current consensus in the community that learned representations have significantly outperformed traditional pre-defined features or filters as the training data expands. I do understand the scientific value of revisiting and challenging that belief as commented by R1, but in order to provoke meaningful discussion, experiments on large-scale dataset like ImageNet are definitely necessary. For these reasons, I think the paper is not ready for publication at ICLR and would like to recommend rejection.
Decision: Reject