A NEW POINTWISE CONVOLUTION IN DEEP NEURAL NETWORKS THROUGH EXTREMELY FAST AND NON PARAMETRIC TRANSFORMS

1 INTRODUCTION . Large Convolutional Neural Networks ( CNNs ) ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; He et al. , 2016 ; Szegedy et al. , 2016b ; a ) and automatic Neural Architecture Search ( NAS ) based networks ( Zoph et al. , 2018 ; Liu et al. , 2018 ; Real et al. , 2018 ) have evolved to show remarkable accuracy on various tasks such as image classification ( Deng et al. , 2009 ; Krizhevsky & Hinton , 2009 ) , object detection ( Lin et al. , 2014 ) , benefited from huge amount of learnable parameters and computations . However , these large number of weights and high computational cost enabled only limited applications for mobile devices that require the constraint on memory space being low as well as for devices that require real-time computations ( Canziani et al. , 2016 ) . With regard to solving these problems , Howard et al . ( 2017 ) ; Sandler et al . ( 2018 ) ; Zhang et al . ( 2017b ) ; Ma et al . ( 2018 ) proposed parameter and computation efficient blocks while maintaining almost same accuracy compared to other heavy CNN models . All of these blocks utilized depthwise separable convolution , which deconstructed the standard convolution with the ( 3 × 3 × C ) size for each kernel into spatial information specific depthwise convolution ( 3 × 3 × 1 ) and channel information specific pointwise ( 1 × 1 × C ) convolution . The depthwise separable convolution achieved comparable accuracy compared to standard spatial convolution with hugely reduced parameters and FLOPs . These reduced resource requirements made the depthwise separable convolution as well as pointwise convolution ( PC ) more widely used in modern CNN architectures . Nevertheless , we point out that the existing PC layer is still computationally expensive and occupies a lot of proportion in the number of weight parameters ( Howard et al. , 2017 ) . Although the demand toward PC layer has been and will be growing exponentially in modern neural network architectures , there has been a little research on improving the naive structure of itself . Therefore , this paper proposes a new PC layer formulated by non-parametric and extremely fast conventional transforms . Conventional transforms that we applied on CNN models are Discrete Walsh-Hadamard Transform ( DWHT ) and Discrete Cosine Transform ( DCT ) , which have widely been used in image processing but rarely been applied in CNNs ( Ghosh & Chellappa , 2016 ) . We empirically found that although both of these transforms do not require any learnable parameters at all , they show the sufficient ability to capture the cross-channel correlations . This non-parametric property enables our proposed CNN models to be significantly compressed in terms of the number of parameters , leading to get the advantages ( i.e . efficient distributed training , less communication between server and clients ) referred by Iandola et al . ( 2016 ) . We note that especially DWHT is considered to be a good replacement of the conventional PC layer , as it requires no floating point multiplications but only additions and subtractions by which the computation overheads of PC layers can significantly be reduced . Furthermore , DWHT can take a strong advantage of its fast version where the computation complexity of the floating point operations is reduced from O ( n2 ) to O ( n log n ) . These non-parametric and low computational properties construct extremely efficient neural network from the perspective of parameter and computation as well as enjoying accuracy gain . Our contributions are summarized as follows : • We propose a new PC layer formulated with conventional transforms which do not require any learnable parameters as well as significantly reducing the number of floating point operations compared to the existing PC layer . • The great benefits of using the bases of existing transforms come from their fast versions , which drastically decrease computation complexity in neural networks without degrading accuracy . • We found that applying ReLU after conventional transforms discards important information extracted , leading to significant drop in accuracy . Based on this finding , we propose the optimal computation block for conventional transforms . • We also found that the conventional transforms can effectively be used especially for extracting high-level features in neural networks . Based on this , we propose a new transformbased neural network architecture . Specifically , using DWHT , our proposed method yields 1.49 % accuracy gain as well as 79.4 % and 49.4 % reduced parameters and FLOPs , respectively , compared with its baseline model ( MobileNet-V1 ) on CIFAR 100 dataset . 2 RELATED WORK . 2.1 DECONSTRUCTION AND DECOMPOSITION OF CONVOLUTIONS . For reducing computation complexity of the existing convolution methods , several approaches of rethinking and deconstructing the naive convolution structures have been proposed . Simonyan & Zisserman ( 2014 ) factorized a large sized kernel ( e.g . 5 × 5 ) in a convolution layer into several small size kernels ( e.g . 3 × 3 ) with several convolution layers . Jeon & Kim ( 2017 ) pointed out the limitation of existing convolution that it has the fixed receptive field . Consequently , they introduced learnable spatial displacement parameters , showing flexibility of convolution . Based on Jeon & Kim ( 2017 ) , Jeon & Kim ( 2018 ) proved that the standard convolution can be deconstructed as a single PC layer with the spatially shifted channels . Based on that , they proposed a very efficient convolution layer , namely active shift layer , by replacing spatial convolutions with shift operations . It is worth noting that the existing PC layer takes the huge proportion of computation and the number of weight parameters in modern lightweight CNN models ( Howard et al. , 2017 ; Sandler et al. , 2018 ; Ma et al. , 2018 ) . Specifically , MobileNet-V1 ( Howard et al. , 2017 ) requires 94 % , 74 % of the overall computational cost and the overall number of weight parameters for the existing PC layer , respectively . Therefore , there were attempts to reduce computation complexity of PC layer . Zhang et al . ( 2017b ) proposed ShuffleNet-V1 where the features are decomposed into several groups over channels and PC operation was conducted for each group , thus reducing the number of weight parameters and FLOPs by the number of groups G. However , it was proved in Ma et al . ( 2018 ) that the memory access cost increases as G increases , leading to slower inference speed . Similarly to the aforementioned methods , our work is to reduce computation complexity and the number of weight parameters in a convolution layer . However , our objective is more oriented on finding out mathe- matically efficient algorithms that make the weights in convolution kernels more effective in feature representation as well as efficient in terms of computation . 2.2 QUANTIZATION . Quantization in neural networks reduced the number of bits utilized to represent the weights and/or activations . Vanhoucke et al . ( 2011 ) applied 8-bit quantization on weight parameters , which enabled considerable speed-up with small drop of accuracy . Gupta et al . ( 2015 ) applied 16-bit fixed point representation with stochastic rounding . Based on Han et al . ( 2015b ) which pruned the unimportant weight connections through thresholding the values of weight , Han et al . ( 2015a ) successfully combined the pruning with 8 bits or less quantization and huffman encoding . The extreme case of quantized networks was evolved from Courbariaux et al . ( 2015 ) , which approximated weights with the binary ( +1 , −1 ) values . From the milestone of Courbariaux et al . ( 2015 ) , Courbariaux & Bengio ( 2016 ) ; Hubara et al . ( 2016 ) constructed Binarized Neural Networks which either stochastically or deterministically binarize the real value weights and activations . These binarized weights and activations lead to significantly reduced run-time by replacing floating point multiplications with 1-bit XNOR operations . Based on Binarized Neural Networks ( Courbariaux & Bengio , 2016 ; Hubara et al. , 2016 ) , Local Binary CNN ( Juefei-Xu et al. , 2016 ) proposed a convolution module that utilizes binarized nonlearnable weights in spatial convolution based on Local Binary Patterns , thus replacing multiplications with addition/subtraction operations in spatial convolution . However , they did not consider reducing computation complexity in PC layer and remained the weights of PC layer learnable floating point variables . Our work shares the similarity to Local Binary CNN ( Juefei-Xu et al. , 2016 ) in using binary fixed weight values . However , Local Binary Patterns have some limitations for being applied in CNN since they can only be used in spatial convolution as well as there are no approaches that enable fast computation of them . 2.3 CONVENTIONAL TRANSFORMS . In general , several transform techniques have been applied for image processing . Discrete Cosine Transform ( DCT ) has been used as a powerful feature extractor ( Dabbaghchian et al. , 2010 ) . For N -point input sequence , the basis kernel of DCT is defined as a list of cosine values as below : Cm = [ cos ( ( 2x+ 1 ) mπ 2N ) ] , 0 ≤ x ≤ N − 1 ( 1 ) where m is the index of a basis and captures higher frequency information in the input signal as m increases . This property led DCT to be widely applied in image/video compression techniques that emphasize the powers of image signals in low frequency regions ( Rao & Yip , 2014 ) . Discrete Walsh Hadamard Transform ( DWHT ) is a very fast and efficient transform by using only +1 and −1 elements in kernels . These binary elements in kernels allow DWHT to perform without any multiplication operations but addition/subtraction operations . Therefore , DWHT has been widely used for fast feature extraction in many practical applications , such as texture image segmentation ( Vard et al. , 2011 ) , face recognition ( Hassan et al. , 2007 ) , and video shot boundary detection ( G. & S. , 2014 ) . Further , DWHT can take advantage of a structured-wiring-based fast algorithm ( Algorithm 1 ) as well as allowing very high efficiency in encoding the spatial information ( Pratt et al. , 1969 ) . The basis kernel matrix of DWHT is defined using the previous kernel matrix as below : HD = ( HD−1 HD−1 HD−1 −HD−1 ) , ( 2 ) where H0 = 1 and D ≥ 1 . In this paper we denote HDm as the m-th row vector of HD in Eq . 2 . Additionally , we adopt fast DWHT algorithm to reduce computation complexity of PC layer in neural networks , resulting in an extremely fast and efficient neural network . 3 METHOD . We propose a new PC layer which is computed with conventional transforms . The conventional PC layer can be formulated as follows : Zijm =W > m ·Xij , 1 ≤ m ≤M ( 3 ) where ( i , j ) is a spatial index , and m is output channel index . In Eq . 3 , N and M are the number of input and output channels , respectively . Xij ∈ RN is a vector of input X at the spatial index ( i , j ) , Wm ∈ RN is a vector of m-th weight W in Eq . 3 . For simplicity , the stride is set as 1 and the bias is omitted in Eq . 3 . Our proposed method is to replace the learnable parameters Wm with the bases in the conventional transforms . For example , replacing Wm with HDm in Eq . 3 , we now can formulate the new multiplication-free PC layer using DWHT . Similarly , the DCT basis kernels Cm in Eq . 1 can substitute for Wm in Eq . 3 , formulating another new PC layer using DCT . Note that the normalization factors in the conventional transforms are not applied in the proposed PC layer , because Batch Normalization ( Ioffe & Szegedy , 2015 ) performs a normalization and a linear transform which can be viewed as a normalization in the existing transforms . The most important benefit of the proposed method comes from the fact that the fast algorithms of the existing transforms can be applied in the proposed PC layers for further reduction of computation . Directly applying above new PC layer gives computational complexity of O ( N2 ) . Adopting the fast algorithms , we can significantly reduce the computation complexity of PC layer fromO ( N2 ) to O ( NlogN ) without any change of the computation results . We demonstrate the pseudo-code of our proposed fast PC layer using DWHT in Algorithm 1 based on the Fast DWHT structure shown in Figure 1a . In Algorithm 1 , for logN iterations , the evenindexed channels and odd-indexed channels are added and subtracted in element-wise manner , respectively . The resulting elements which were added and subtracted are placed in the first N/2 elements and the last N/2 elements of the input of next iteration , respectively . In this computation process , each iteration requires only N operations of additions or subtractions . Consequently , Algorithm 1 gives us complexity of O ( NlogN ) in addition or subtraction . Compared to the existing PC layer that requires complexity of O ( N2 ) in multiplication , our method is extremely cheaper than the conventional PC layer in terms of computation costs as shown in Figure 1b and in power consumption of computing devices ( Horowitz , 2014 ) . Note that , similarly to fast DWHT , DCT can also be computed in a fast manner that recursively decomposes the N -point input sequence into two subproblems of N/2-point DCT ( Kok , 1997 ) . Compared to DWHT , DCT takes advantage of using more natural shapes of cosine basis kernels , which tend to provide better feature extraction performance through capturing the frequency information . However , DCT inevitably needs multiplications for inner product betweenC andX vectors , and a look up table ( LUT ) for computing cosine kernel bases which can increase the processing time and memory access . On the other hand , as mentioned , the kernels of DWHT consist only of +1 , −1 which allows for building a multiplication-free module . Furthermore , any memory access towards kernel bases is not needed if our structured-wiring-based fast DWHT algorithm ( Algorithm 1 ) is applied . Our comprehensive experiments in Section 3.1 and 3.2 show that DWHT is more efficient than DCT in being applied in PC layer in terms of trade-off between the complexity of computation cost and accuracy . Note that , for securing more general formulation of our newly defined PC layer , we padded zeros along the channel axis if the number of input channels is less than that of output channels while truncating the output channels when the number of output channels shrink compared to that of input channels as shown in Algorithm 1 . Figure 1a shows the architecture of fast DWHT algorithm described in Algorithm 1 . This structuredwiring-based architecture ensures that the receptive field of each output channels isN , which means each output channel is fully reflected against all input channels through log2N iterations . This fullyreflected property helps to capture the input channel correlations in spite of the computation process of what channel elements will be added and subtracted being structured in a deterministic manner . For successfully fusing our new PC layer into neural networks , we explored two themes : i ) an optimal block search for the proposed PC ; ii ) an optimal insertion strategy of the proposed block found by i ) , in a hierarchical manner on the blocks of networks . We assumed that there are an optimal block unit structure and an optimal hierarchy level ( high- , middle- , low-level ) blocks in the neural networks favored by these non-learnable transforms . Therefore , we conducted the experiments for the two aforementioned themes accordingly . We evaluated the effectiveness for each of our networks by accuracy fluctuation as the number of learnable weight parameters or FLOPs changes . For comparison , we counted total FLOPs with summation of the number of multiplications , additions and subtractions performed during the inference . Unless mentioned , we followed the default experimental setting as 128 batch size , 200 training epochs , 0.1 initial learning rate where 0.94 is multiplied per 2 epochs , and 0.9 momentum with 5e-4 weight decay value . In all the experiments , the model accuracy was obtained by taking an average of Top-1 accuracy values from three independent training results . Algorithm 1 A new pointwise convolution using fast DWHT algorithm Input : 4D input features X ( B ×N ×H ×W ) , output channel M 1 : n← log2N 2 : if N < M then 3 : ZeroPad1D ( X , axis=1 ) . pad zeros along the channel axis 4 : end if 5 : for i← 1 to n do 6 : e← X [ : , : : 2 , : , : ] 7 : o← X [ : , 1 : : 2 , : , : ] 8 : X [ : , : N/2 , : , : ] ← e+ o 9 : X [ : , N/2 : , : , : ] ← e− o 10 : end for 11 : if N > M then 12 : X ← X [ : , : M , : , : ] 13 : end if

Summary: This paper proposes a new pointwise convolution layer, which is non-parametric and can be efficient thanks to the fast conventional transforms. Specifically, it could use either DCT or DHWT to do the transforming job and explores the optimal block structure to use this new kind of PC layer. Extensive experimental studies are provided to verify the new PC layer and experimental results show that the new layer could reduce the parameters and FLOPs while not loosing accuracy.