Learning Features with Parameter-Free Layers

1 INTRODUCTION . Image classification has been advanced with deep convolutional neural networks ( Simonyan & Zisserman , 2015 ; Huang et al. , 2017 ; He et al. , 2016b ) with the common design paradigm of the network building blocks with trainable spatial convolutions inside . Such trainable layers with learnable parameters effectively grasp attentive signals to distinguish input but are computationally heavy . Rather than applying pruning or distillation techniques to reduce the computational cost , developing new efficient operations has been another underlying strategy . For example , a variant of the regular convolution , such as depthwise convolution ( Howard et al. , 2017 ) has been proposed to bring more efficiency by reducing the inter-channel computations . The operation has benefits in the computational budgets , including the number of parameters and FLOPs . However , the networks heavily using the depthwise convolution ( Howard et al. , 2017 ; Sandler et al. , 2018 ; Tan & Le , 2019 ) have an inherent downside of the latency , which generally do not reach the speed of the regular convolution . In a line of study of efficient operations , there have been many works ( Wang et al. , 2018 ; Wu et al. , 2018 ; Han et al. , 2020 ; Tan & Le , 2021 ) based on the regular convolution and the depthwise convolution . Most methods utilize the depthwise convolution ’ s efficiency or target FLOPs-efficiency but are slow in the computation . Meanwhile , parameter-free operations were proposed ; a representative work is the Shift operation ( Wu et al. , 2018 ) . Its efficiency stems from the novel operation without learning spatial parameters by letting the feature mixing convolutions learn from the shifted features . However , the implementation does not bring about the actual speed-up as expected . This is because the operation-level optimization is still demanding compared to the regular convolution with highly optimized performance . Another parameter-free operation feature shuffling ( Zhang et al. , 2018 ) is a seminal operation that reduces the computational cost of the feature mixing layer . However , it hardly plays the role of a spatial operation . In this paper , we focus on efficient parameter-free operations that actually replace trainable layers for network design . We revisit the popular parameter-free operations , the max-pool and the avg-pool op- erations ( i.e. , layers ) , which are used in many deep neural networks ( Simonyan & Zisserman , 2015 ; Huang et al. , 2017 ) restricted to perform downsampling ( Simonyan & Zisserman , 2015 ; Howard et al. , 2017 ; Sandler et al. , 2018 ; He et al. , 2016b ) . Can those simple parameter-free operations be used as the main network building block ? If so , one can reduce a large portion of the parameters and the overall computational budget required during training and inference . To answer the question , the max-pool and the avg-pool operations are chosen to be demonstrated as representative simple parameter-free operations . We first conduct comprehensive studies on the layer replacements of the regular convolutions inside networks searched upon the baseline models with model training . Additionally , we incorporate a neural architecture search ( Liu et al. , 2019 ) to explore effective architectures based on the operation list with the parameter-free operations and convolutional operations . Based on the investigations , we provide a simple rule of thumb to design an efficient architecture using the parameter-free operations upon primitive network architectures . The design guide is applied to popular heavy networks and validated by the performance trained on ImageNet ( Russakovsky et al. , 2015 ) . It turns out that our models have apparent benefits to the computational costs , particularly the faster model speeds . In addition , ImageNet-C ( Hendrycks & Dietterich , 2019 ) and ImageNet-O ( Hendrycks et al. , 2021 ) results show our models are less prone to be overfitting . We further propose a novel deformable parameter-free operation based on the max-pool and the avg-pool to demonstrate the future of a parameter-free operation . Finally , we show that the parameter-free operations can successfully replace the self-attention layer ( Vaswani et al. , 2017 ) , thus attaining further efficiency and speed up . We summarize our contributions as follows : • We study whether parameter-free operations can replace trainable layers as a network building block . To our knowledge , this is the first attempt to investigate a simple , builtin parameter-free layer as a building block for further efficiency ( §3 ) . • We provide a rule of thumb for designing a deep neural network including convolutional neural networks and vision transformers with parameter-free layers ( §4 ) . • Experimental results show that our efficient models outperform the previous efficient models and yield faster model speeds with further robustness ( §5 ) . 2 PRELIMINARIES . A network building block where trainable layers are inside is a fundamental element of modularized networks ( Xie et al. , 2017 ; Sandler et al. , 2018 ; Tan & Le , 2019 ) . We start a discussion with the elementary building block and then explore more efficient ones . 2.1 BASIC BUILDING BLOCKS . Convolution Layer . We recall the regular convolutional operation by formulating with matrix multiplication first . Let f ∈ Rcin×H×W as the input feature , the regular convolution of the feature f with kernel size k and stride r is given as yo , i , j = σ bk/2c∑ h , w=−bk/2c cin∑ u=1 Wo , u , h , w · fu , r∗i+h , r∗j+w , ( 1 ) where W denotes the weight matrix , and the function σ ( · ) denotes an activation function such as ReLU ( Nair & Hinton , 2010 ) with or without batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) . The convolution itself has been a building block due to the expressiveness and design flexibility . VGG ( Simonyan & Zisserman , 2015 ) network was designed by accumulating the 3×3 convolution to substitute the convolutions with larger kernel sizes but still have high computational costs . Bottleneck Block . We now recall the bottleneck block , which primarily aimed at efficiency . We represent the bottleneck block by the matrix multiplication as equation 1 : yo , i , j = σ ρcin∑ v=1 Po , v · σ bk/2c∑ h , w=−bk/2c ρcin∑ u=1 Wv , u , h , w · gv , r∗i+h , r∗j+w , ( 2 ) where go , i , j = σ ( ∑cin u=1Qo , u · fu , i , j ) , and the matrix P and Q denote the weights of 1×1 convolutions with the inner dimensions ρcin . This design regime is efficient in terms of the computational budgets and even proven to be effective in the generalization ability when stacking up the bottleneck blocks compared with the basic blocks . Albeit a 3×3 convolution is replaced with two feature mixing layers ( i.e. , 1×1 convolutions ) , the expressiveness is still high enough with a low channel expansion ratio ρ ( i.e. , ρ = 1/4 in ResNet ( He et al. , 2016b ; Xie et al. , 2017 ) ) . However , due to the presence of the regular 3×3 convolution , only adjusting ρ hardly achieves further efficiency . 2.2 EFFICIENT BUILDING BLOCKS . Inverted Bottleneck . The grouped operations , including the group convolution ( Xie et al. , 2017 ) and the depthwise convolution ( Howard et al. , 2017 ) ) have emerged as more efficient building blocks . Using the depthwise convolution inside a bottleneck ( Sandler et al. , 2018 ) is called the inverted bottleneck , which is represented as yo , i , j = σ ρcin∑ v=1 Po , v · σ bk/2c∑ h , w=−bk/2c Wv , h , w · gv , r∗i+h , r∗j+w , ( 3 ) where the summation over the channels in equation 2 has vanished . This slight modification takes advantage of further generalization ability , and therefore stacking of the blocks leads to outperform ResNet with a better trade-off between accuracy and efficiency as proven in many architectures ( Tan & Le , 2019 ; Han et al. , 2021 ) . The feature mixing operation in equation 2 with the following pointwise activation function ( i.e. , ReLU ) may offer a sufficient rank of the feature with the efficient operation . However , the inverted bottleneck and the variants below usually need a large expansion ratio ρ > 1 to secure the expressiveness ( Sandler et al. , 2018 ; Howard et al. , 2019 ; Tan & Le , 2019 ) , so the actual speed is hampered by the grouped operation that requires more optimization on GPU ( Gibson et al. , 2020 ; Lu et al. , 2021 ) . Variants of Inverted Bottleneck . More refinements on the bottleneck block could bring further efficiency ; the prior works ( Wang et al. , 2018 ; Han et al. , 2020 ; Tan & Le , 2021 ; Wu et al. , 2018 ) fully or partially redesigned the layers in the bottleneck with new operations , and therefore theoretical computational costs have been decreased . VersatileNet ( Wang et al. , 2018 ) replaced the convolutions with the proposed filters consisting of multiple convolutional filters ; GhostNet ( Han et al. , 2020 ) similarly replaced the layers with the proposed module that concatenates a preceding regular convolution and additional depthwise convolution . Rather than involving new operations , a simple replacement by simplifying the operations has been proposed instead . EfficientNetV2 ( Tan & Le , 2021 ) improved the training speed by fusing the pointwise and the depthwise convolutions to a single regular convolution . This takes advantage of using the highly-optimized operation rather than using new unoptimized operations . ShiftNet ( Wu et al. , 2018 ) simplify the depthwise convolution to the shift operation , where the formulation is very similar to equation 3 : yo , i , j = σ ρcin∑ v=1 Po , v · σ bk/2c∑ h , w=−bk/2c Wv , h , w · gv , r∗i+h , r∗j+w , ( 4 ) where Wv , : , : is simplified to have one 1 and 0 for the rest all h , w for each v. This change involves a new operation , so-called Shift , that looks like shifting the features after the computation , and the preceding and the following pointwise convoltuions mix the shifted output . The shift operation is actually classified as a parameter-free operation , but a large expansion ratio is yet needed to hold the expressiveness . Furthermore , the implementation is not as readily optimized as regular operations , even in CUDA implementation ( He et al. , 2019 ; Chen et al. , 2019a ) . 3 EFFICIENT BUILDING BLOCK WITH PARAMETER-FREE OPERATIONS . In this section , we extend an efficient building block by incorporating parameter-free operations . We empirically study whether the parameter-free operations can be used as the main building block in deep neural networks like regular convolution . 3.1 MOTIVATION . The learning mechanism of ResNets ( He et al. , 2016b ; a ) has been analyzed as the iterative unrolled estimation ( Greff et al. , 2017 ; Jastrzebski et al. , 2018 ) which iteratively refine the features owing to the presence of skip connections . This indicates that some layers do not contribute much to learning , which has also been revealed in layer-drop experiments ( Veit et al. , 2016 ; Huang et al. , 2016 ) . In this light , we argue some building blocks in a residual network can be replaced with parameterfree operations . We investigate the replacement of the spatial operation in popular networks with parameter-free operations to rethink the common practice of network design .

Summary: This paper focuses on designing efficient deep networks with a proposed novel parameter-free layers. Drawing the spirit of depth-wise convolution, shift operation, the paper studies the ways of inserting parameter-free operations into the building block using NAS technique. The resulted models perform both effectively and efficiently on ImageNet dataset.