Learning Features with Parameter-Free Layers
Authors: Dongyoon Han
YoungJoon Yoo
Beomyoung Kim
Byeongho Heo
Venue: ICLR 2022 Poster (2022)
Keywords: ImageNet
efficient network architecture
network design
image classification

Abstract: Trainable layers such as convolutional building blocks are the standard network design choices by learning parameters to capture the global context through successive spatial operations. When designing an efficient network, trainable layers such as the depthwise convolution is the source of efficiency in the number of parameters and FLOPs, but there was little improvement to the model speed in practice. This paper argues that simple built-in parameter-free operations can be a favorable alternative to the efficient trainable layers replacing spatial operations in a network architecture. We aim to break the stereotype of organizing the spatial operations of building blocks into trainable layers. Extensive experimental analyses based on layer-level studies with fully-trained models and neural architecture searches are provided to investigate whether parameter-free operations such as the max-pool are functional. The studies eventually give us a simple yet effective idea for redesigning network architectures, where the parameter-free operations are heavily used as the main building block without sacrificing the model accuracy as much. Experimental results on the ImageNet dataset demonstrate that the network architectures with parameter-free operations could enjoy the advantages of further efficiency in terms of model speed, the number of the parameters, and FLOPs. Code and ImageNet pretrained models are available at https://github.com/naver-ai/PfLayer.



Published as a conference paper at ICLR 2022
LEARNING FEATURES WITH PARAMETER-FREB LAYERS
Dongyoon Han', YoungJoon Yoo!?, Beomyoung Kim², Byeongho Heo NAVER Al Lab, ?NAERR CLOVV
ABSTRACT
Trainable layers such convolutional building blocks the standard network design choices by learning parameters to capture the global context through suc- cessive spatial operations. When designing an efficient network, trainable lay- ers such as the depthwise convolution is the source of efficiency in the number of parameters and FLOPs, but there was little improvement to the model speed in practice. This paper argues that simple built-in parameter- free operations can be a favorable alternative the efficient trainable layers replacing spatial oper- ations in network architecture. We aim to break the stereotype of organizing the spatial operations of building blocks into trainable layers. Extensive experi- mental analyses based on layer-level studies with fully-trained models and neural architecture searches are provided to investigate whether parameter-free opera- tions such as the max-pool are functional. The studies eventually give simple yet effective idea for redesigning network architectures, where the parameter- free operations are heavily used as the main building block without sacrificing the model accuracy as much. Experimental results on the ImageNet dataset demon- strate that the network architectures with parameter-free operations could enjoy the advantages of further efficiency in terms of model speed, the number of the parameters, and FLOPs. Code and ImageNet pretrained models are available pettss:/giie com/ http//vveerraaaaaa haver-ai/Pfaate aiffaaeer...
INTRODUCTION
Image classification has been advanced with deep convolutional neural networks (Simonyan & 7is- serman, 2015; Huang al., 2017: He al., 2016b) with the common design paradigm of the net- work building blocks with trainable spatial convolutions inside. Such trainable layers with learnable parameters effectively grasp attentive signals to distinguish input but are computationally heavy. Rather than applying pruning distillation techniques reduce the computational cost, developing new efficient operations has been another underlying strategy. For example, a variant of the regu- lar convolution, such depthwise convolution (Howard et al., 2017) has been proposed to bring more efficiency by reducing the inter-channel computations. The operation has benefits in the com- putational budgets, including the number of parameters and FLOPs. However, the networks heavily using the depthwise convolution (Howard et al., 2017; Sandler et al., 2018; Tan & Le, 2019) have an inherent downside of the latency, which generally do not reach the speed of the regular convolution.
In a line of study of efficient operations, there have been many works (Wang et al., 2018; Wu et al., 2018; Han et al., 2020; Tan Le, 2021) based on the regular convolution and the depthwise convo- lution. Most methods utilize the depthwise convolution's 's efficiency or target FLOPs- PLOOs-effiiieccy but are slow in the computation. Meanwhile, parameter- free operations were proposed; a representative work is the Shift operation (Wu et al., 2018). Its efficiency stems from the novel operation without learning spatial parameters by letting the feature mixing convolutions learn from the shifted features. However, the implementation does not bring about the actual speed-up expected. This is because the operation-lee level optimization still demanding compared to the regular convolution with highly optimized performance. Another parameter- free operation feature shuffling (Zhang et al., 2018) is seminal operation that reduces the computational cost of the feature mixing layer. However, it hardly plays the role of spatial operation.
In this paper, we focus on efficient parameter-Fa free operations that actually replace trainable layers for network design. We revisit the popular marameter-rree operations, the max-pool and the avg-pool op-

---

Published as a conference paper at ICLR 2022
erations (i.e., layers), which are used in many deep neural networks (Simonyan & Zisserman, 2015; Huang et al., 2017) restricted to perform downsampling (Simonyan & Zisserman, 2015; Howard et al., 2017; Sandler et al., 2018; He et al., 2016b). Can those simple parameter-free operations be used as the main network building block? If so, one can reduce a large portion of the param- eters and the overall computational budget required during training and inference. To answer the question, the max-pool and the avg-pool operations are chosen to be demonstrated as representative simple parameter-free operations. We first conduct comprehensive studies on the layer replacements of the regular convolutions inside networks searched upon the baseline models with model train- ing. Additionally, we incorporate a neural architecture search (Liu et al., 2019) to explore effective architectures based on the operation list with the parameter-free operations and convolutional oper- ations. Based on the investigations, we provide a simple rule of thumb to design an efficient archi- tecture using the parameter-free operations upon primitive network architectures. The design guide is applied to popular heavy networks and validated by the performance trained on ImageNet (Rus- sakovsky et al., 2015). It turns out that our models have apparent benefits to the computational costs, particularly the faster model speeds. In addition, ImageNet-C (Hendrycks & Dietterich, 2019) and ImageNet-O (Hendrycks al., 2021) results show models are less prone to be overfitting. We further propose novel deformable parameter-free operation based on the max-pool and the avg-pool demonstrate the future of parameter-free operation. Finally, show that the parameter-free op- erations can successfully replace the self-attention layer (Vaswani et al., 2017), thus attaining further efficiency and speed up. We summarize our contributions as follows:
We study whether parameter-free operations can replace trainable layers as a network building block. To our knowledge, this is the first attempt to investigate a simple, built- in parameter- free layer as building block for further efficiency ($3).
We provide rule of thumb for designing deep neural network including convolutional neural networks and vision transformers with parameter-fiee layers ($4).
Experimental results show that our efficient models outperform the previous efficient mod- els and yield faster model speeds with further robustness ($5).
2 PRELIMINARIES
A network building block where trainable layers are inside is fundamental element of modularized networks (Xie et al., 2017; Sandler et al., 2018; Tan & Le, 2019). We start a discussion with the elementary building block and then explore more efficient ones.
2.1 BASIC BUILDING BLOCKS
Convolution Layer. We recall the regular convolutional operation by formulating with matrix multiplication first. Let f E x as the input feature, the regular convolution of the feature with kernel size k and stride r is given
[k/2]
Can
Yo.i.j a
Σ
Σ
Wo Surreith
W,
1k/2
where W denotes the weight matrix, and the function af-) denotes an activation function such as ReLU (Nair & Hinton, 2010) with or without batch normalization (BN) (Ioffe & Szegedy, 2015). The convolution itself has been a building block due to the expressiveness and design flexibility. VGG (Simonyan & Zisserman, 2015) network was designed by accumulating the 3x3 convolution to substitute the convolutions with larger kernel sizes but still have high computational costs.
Bottleneck Block. We now recall the bottleneck block, which primarily aimed at efficiency. We represent the bottleneck block by the matrix multiplication as equation 1:
pein Σ Po.v
k/2J Σ
pein
Yo,k.J
We,uhhw Iurretthrrstu
(2)
h,== k/2)
==1 Qonu fus,,), and the matrix P and Q denote the weights Ixl convolu- tions with the inner dimensions pCan- This design regime is efficient in terms of the computational budgets and even proven to be effective in the generalization ability when stacking up the bottle- neck blocks compared with the basic blocks. Albeit 3x3 convolution is replaced with two feature
where Go.i.j = o (E Gin
2

---

Published as a conference paper at ICLR 2022
mixing layers (i.e., 1x1 convolutions), the expressiveness is still high enough with low channel expansion ratio p (i.e., = 1/4 /44 in ResNet (He et al., 2016b; Xie et al., 2017)). However, due to the presence of the regular x3 convolution, only adjusting p hardly achieves further efficiency.
2.2 EFFICIENT BUILDING BLOCKS
Inverted Bottleneck. The grouped operations, including the group convolution (Xie et al., 2017) and the depthwise convolution (Howard et al., 2017)) have emerged as more efficient building blocks. Using the depthwise convolution inside bottleneck (Sandler et al., 2018) is called the inverted bottleneck, which is represented as
pein Σ PO.
[k/2]
)
Yo,ij
Σ
W Wo,h.w Iorraithraatti
h,-=- [k/2]
where the summation over the channels in equation has vanished. This slight modification takes advantage of further generalization ability, and therefore stacking of the blocks leads to outperform ResNet with a better trade-off between accuracy and efficiency as proven in many architectures (Tan & Le, 2019; Han et al., 2021). The feature mixing operation in equation with the following point- wise activation function (i.e., ReLU) may offer sufficient rank of the feature with the efficient operation. However, the inverted bottleneck and the variants below usually need large expansion ratio > 1 to secure the expressiveness (Sandler et al., 2018; Howard et al., 2019; Tan & Le, 2019), so the actual speed hampered by the grouped operation that requires more optimization on GPU (Gibson et al., 2020; Lu et al., 2021).
Variants of Inverted Bottleneck. More refinements on the bottleneck block could bring further efficiency; the prior works (Wang al., 2018; Han al., 2020; Tan & Le, 2021; Wu et al., 2018) fully or partially redesigned the layers in the bottleneck with new operations, and therefore the- oretical computational costs have been decreased. VersatileNet (Wang et al., 2018) replaced the convolutions with the proposed filters consisting of multiple convolutional filters; GhostNet (Han al., 2020) similarly replaced the layers with the proposed module that concatenates a preceding regular convolution and additional depthwise convolution. Rather than involving new operations, simple replacement by simplifying the operations has been proposed instead. FffficienNNet22 (Tan & Le, 2021) improved the training speed by fusing the pointwise and the depthwise convolutions to single regular convolution. This takes advantage of using the highly-optimized operation rather than using new unoptimized operations. ShiftNet (Wu et al., 2018) simplify the depthwise convolution to the shift operation, where the formulation is very similar to equation 3:
pein Σ Po.
[k/2] Σ
Yori.
W Wothow :m Jorretth
where W Wo..: is simplified to have one 1 and O for the rest all h, w for each v. This change involves new operation, so-called Shift, that looks like shifting the features after the computation, and the preceding and the following pointwise convolutions mix the shifted output. The shift operation is actually classified as a parameter-fre operation, but large expansion ratio yet needed to hold the expressiveness. Furthermore, the implementation is not readily optimized as regular operations, even CUDA implementation (He et al., 2019; Chen et al., 2019a).
EFFICIENT BUILDING BLOCK WITH PARAMETER- FREE OPERATIONS
In this section, we extend an efficient building block by incorporating parameter-free free operations. We empirically study whether the parameter-free operations can be used as the main building block in deep neural networks like regular convolution.
3.1 MOTIVATION
The learning mechanism of ResNets (He et al., 2016b;a) has been analyzed the iterative unrolled estimation (Greff et al., 2017; Jastrzebski et al., 2018) which iteratively refine the features owing to the presence of skip connections. This indicates that some layers do not contribute much to learning. which has also been revealed in layer-drop experiments (Veit et al., 2016; Huang et al., 2016). In this light, we argue some building blocks in a residual network can be replaced with parameter- free operations. We investigate the replacement of the spatial operation in popular networks with parameter- free operations rethink the common practice of network design.
3

---

Published as a conference paper at ICLR 2022
SGD
AdamW
AdamP
2.0g
Widths: 32
12.
of aroosms
Top- acc. (9%
12.6
Channel
64-
57.
2.4
Top-
Cotlv
cony
dwconv
42:5
deccoivv
max
2,00
0.30
100
2002
Channel Exp. Ratio
Channel Exp. Ratio
Charnel Exp. Ratio
parareem.. (89)
Widths: 64
Top-1 lcce 196
72.3
+1.32
3.4
Channel
1.22
3.2
22.5
0.
100
210
3.2
65.0
3.50
1.00
2.00
2.2 2.50
1.00
max
2.50
4.6
67.5
2.2 Channel Exp. Ratio
max
2.00
Channel Exp. Ratio
Chanmel Exp. Ratio
Figure 1. Single bottleneck study. We visualize top-1 accuracy (solid lines) trained with the different setups including 1) varying channel expansion ratios inside a bottleneck; 2) the different channel widths: 32 (up- per row) and 64 (lower row); 3) diverse optimizers: SGD (left), AdamW (middle), and AdamP (right): We further plot accuracy per # parameters (dashed lines) to show the parameter-efficiency of the operations. We observe the regular convolutions work well but be replaccable at low channel expansion ratio; the alternative operations are highly efficient; the max-pool consistently beats the depthwise convolution.
3.2 RETHINKING PARAMETER -REE OPERATIONS
Our goal is to design efficient building block that boosts model speed in practice while maintain- ing model accuracy. Based on equation 2. we remove the inter-channel operation similarly done in equation 3 and equation 4. Then, instead of assigning We,h,w to be single value, in equation 4, we let W have a dependency on the feature g (i.e., W function s()) Then, the layer would have different learning dynamics interacting with P and Q.
by introducing a
Among many candidates to formalize s(.), we allocate the function with a simple one that does not have trainable parameters. We let s(-) pick some large values of g in the range of all h, w per each v like the impulse function. Here, we simplify the function s(-) again to use only the largest values of g. where we have the representation of W .. (h", ..w -argmmaa Tv. and other W.,,,ar to be in equation 3. One may come up with another simple option: let s(-) compute the moments such as mean or variance of In practice, those operations can be replaced with built-in spatial pooling operations, which are highly optimized at the operation level. In the following section, we empirically study the effectivness of efficient parameter-free operations in network designs.
3.3 EMPIRICAL STUDIES
On a Single Bottleneck. Our study begins with a single bottleneck block of ResNet (He et al., 20) We identify such a parameter-free operation can replace the depthwise convolution which is a strong alternative of the regular convolution. We train a large number of different architectures on CIFAR10 10 and observe the accuracy trend of trained models. The models consist of a single bot- tleneck, but the channel expansion ratio inside a bottleneck varies from 0.25 to 2, and there are two options for the base-channel width (32 and 64). Training is done with three different optimizers of SGD, AdamW (Loshchilov & Hutter, 201 (b), and AdamP (Heo et al., 2021a). Finally, we have to- tal 4x2x3x3=72 models for the study. Fig.1 exhibit that the parameter-free operation consistently outperforms the depthwise convolution in a bottleneck The regular convolutions clearly achieved higher absolute accuracy, but the efficiency (the accuracy over the number of parameters) getting worse for larger expansion ratios. We observe that the accuracy gaps between the regular convolu- tion and the parameter-free operation are low for small channel expansion ratios (especially at the 1/4 where the original bottleneck (He et al., 2016b; Xie et al., 2017) adopts). It means that the regular convolution in the original bottleneck block with the expansion ratio of 1/4 can be replaced by the parameter-free free operation.
'The bottleneck with the max-poo pool operation here is designed by following the formulation in $3.2; we call it the efficient bottleneck. Note that the max-pool operation needs to be adopted due the performance failures of using the avg-pool operation (see Appendix B).
4

---

Published as a conference paper at ICLR 2022
32.4
2
22.0
13.4
51.6
10.0
16.3
19.5
0.23
0.21
0.2
51.
28
Speed
Speed
parama (NI
FLOP&
(a) Speed (1-image) (b) Speed (256-images)
(c) parameters
(d) FLOPs
Figure 2. Multiple bottlenecks study. We respectively pick the 20% best-performing models and visualize in the comparison graphs: (a) Accuracy vs. speed with the batch size I; (b) Accuracy vs. speed with the batch size of 256; (c) Accuracy vs. # parameters: (d) Accuracy vs. FLOPs (use FLOPs to mean multiply-adds). The depthwise convolution (blue dots) is an efficient operation in # parameters and FLOPs, but the parameter-free free operation (red dots) has a clear benefit in the model speed in practice. A particular parameter-fiee operation be an alternative to the depthwise convolution when replacing the regular convolutions.
On Multiple Bottlenecks. We extend the single bottleneck study to deeper networks having mul- tiple bottleneck blocks. We similarly study the bottleneck with different spatial operations, includ- ing the depthwise convolution and parameter-free operations, replacing the regular convolution. We choose the max-pool operation again as the parameter-free operation and compare with the depth- wise convolution. To study the impact of repeatedly using parameter-free operations in-depth, we exhaustively perform the layer replacements at layer-level upon the baseline network. We report the trade-off' between accuracy and speed by replacing the regular convolutions with the max- max-pool operation and the depthwise convolution. We use the baseline network, which has eight bottleneck blocks in standard ResNet architecture (i.e., ResNet-22), and the study has done by replacing each block have efficient operations (i.e., the efficient bottleneck and the bottleneck with the depthwise convolution). We train all the networks with large epochs (300 epochs) due to the depthwise convo- lution. Fig.2 2 illustrates the trade-off's of top-1 accuracy and the actual model speed; we observe the models with the parameter-fre operations are faster than those with the depthwise convolution as the model accuracies are comparable.
Table 1. Neural architecture search with individual cell searches. We investigate how parameter-Iree oper- ations are chosen with convolutional operations. Six different normal cells are searched individually, and two reduction cells searched by unified singlc cell. Surprisingly. the parameter- free operations are consis- tently found regardless of the settings. The parameter-fiee operations in each cell are represented as max @ [(d, Vii... x))] and avg @ [(d, Tii.... . x which denote the max- maxx-pool.3x 3x3 and the avg-pool_3xx 3x3 appear at every edge towards œ1 24 th nodes in d-th (from the input) cell, respectively.
# of Nodes Seed No. Normal Cells
Reduction Cell
Prec-1
Conifeed
no parameter-free ops.
avg @1(33 2), 2]1
87.84%
max @ D)]
max @(33, D), (6. D)
86.15%
max [ D)]
max D. (6.
86.27%
max @12,, D)]
max @ 1), (6. D)
85.42%
max ((2 2), (5. 1:2)
purameter-free ops.
87.64%
max 68 [[4, 2), (5, 1:2), (7, 2)], avg @1(7. D)
no parameter-free ops. no parameter- free ops.
87.70%
max [(2, 1), (5, 1:2))
87.65%
max (5. 1:2:3), (7. 1:2:3]
max @ ((33) 3), (6. 3)]
88.09%
max [(4, 2), (5, 2). (7, 2),, avg @[77 D)
max @ 3), (6, 3)]
88.06%
max [(4, 1:3), (5. 3)), avg @((5. 2))
no parameter free ops.
88.03%
max @ [(2, 3;4), (5, 1:2:3:4), (7, 1)J, avg @1 1:20)
avg @I(3, 1:2), (6, 1:2)]
88%
max [(4, 1:3:4), (5, 2:3:4), (7, 1:2,3:4),, avg @ D)
no parameter- free ops.
87.79%
@1 3), (7, 1:2)), avg @[((4 1:2,3), (8. 2)
avg 69[(3, 2). 2)]
87.545)
On Neural Architecture Scarches. We further investigate whether parameter-free operations are likely to be chosen for model accuracy along with trainable operations in a neural architecture search (NAS). We choose DARTS (Liu et al., 2019) with the default architecture of eight cells and perform searches on CIFAR-10 (Krizhevsky, 2009)2. Towards a more sophisticated study, we refine the search configuration individually rather than following the default setting searching one unified normal cell. This is be- cause when searching with the unified normal cell, it is more likely to be searched without the parameter-free operations to secure the expressiveness of the entire network (see the search result
follows. First, we force the entire normal cells be searched
We follow the original training settings of DARTS (Liu et al., 2019). Additionally, to avoid the collapse of DARTS due to the domination of skip connections in learned architectures, we insert the dropout at the SKIP CONNECT operation proposed in the method (Chen al., 2019b).
5

---

Published as a conference paper at ICLR 2022
at the first row in Table 1). Therefore prevent this with the more natural configuration. Second, the operation list is changed to ensure fairness of the expressiveness among the operations; since the primitive DARTS has the default operation list with the separable convolutions (SEP CONV 3x3, SEP CONV 5) and the dilated separable convolutions (DIL. CONV 3 3x3, DII. CONV 5x5) which consist of multiple convolutions and ReLU with BN. Thus, we need to simplify the primitive op- erations to
x3, DVG_POOL_3 x3,, CONV LENV_IxIL 11, CONV.3 x3, DW wc____NV__ x3,3, ZERO, SKIP-C .CONNECT] Finally, we further perform searches with different numbers of nodes inside each cell to confirm the trend of search results regarding the architectural constraint. All the searches are repeated with three different seed numbers.
As shown in Table I and Fig.3, we observe the parameter-fre operations are frequently searched in the normal cells, which is different from the original search result (at the first row) similarly shown in the previous works (Liu et al., 2019; Chen et al., 2019;; Liang et al., 2019; Chu et al., 2020). Moreover, the accuracies are similar to the original setting with two unified cells, where the parameter-free operations are not searched in the normal cell (see the first and the last rows). Interestingly, the number of the picked parameter-free free operations increases as the num- ber of nodes increases; they do not dominate the reduction cells. As a result, it turns out that the accuracy objective in searches lets the parameter-free freee operations become chosen, and therefore the operations can be used at certain positions which looks similar to the Inception blocks (Szegedy et al., 2015; Ioffe & Szegedy, 2015; Szegedy et al., 2016). Notice that similar results are produced with different settings and seeds.
CL
2009
(a) 5-th normal cell
(b) 5-th normal cell
ran pad
dr. 365
(c) 7-th normal cell
(d) 5-th normal cell
Figure 3. Visualization of the searched cells. The ex- ample searched cells shown in Table I are visualized.
4 DESIGNING EFFICIENT DEEP NEURAL NETWORKS
Based on the studies above, parameter- free operations can be a building block of designing network architecture. We apply the parameter-fit operations to redesign deeper neural network architec- ture, including a convoluption neural network (CNN) and a vision transformer (ViT).
4.1 EFFICIENT CNN ARCHITECTURES
Hybrid Architecture with Efficient Bottlenecks. We employ the proposed efficient bottleneck to design deeper efficient CNNs. The max-pool operation plays role of the spatial operation in the efficient bottleneck; specifically, the 3x3 convolution, BN, and ReLU triplet are replaced with max-pool operation To design the entire network architecture, fully utilizing the efficient bottle- necks would yield a much faster inference speed model, but a combination of the efficient bottleneck and the regular bottleneck has better trade-off between accuracy and speed. We use ResNet50 (He et al., 2016b) as the baseline to equip simple heuristic of building a hybrid architecture of the efficient bottleneck and the regular bottleneck. The observation from the NAS searches in $3.3, the freeeeter-free operations are not explored much at the downsampling blocks but normal blocks. Thus, we similarly design a new efficient network architecture using the efficient bottlenecks as the fundamental building block except for the downsampling positions with the regular bottlenecks. We call this model hybrid architecture. The details of the architectures are illustrated in Appendix A.
Architectural Study. The architectural study is further conducted with the replacements of the regular spatial convolutions in the regular bottlenecks. We involve the efficient bottlenecks into the stages of ResNet50, replacing the existing regular bottlenecks exhaustively to cover the design space as much as possible. It worth noting that performing a neural architecture search would yield a pre- cise design guide, but direct search on ImageNet is costly. We use the ImageNet dataset to provide a more convincing result of the investigation. We also report the performance of the competitive
We found that parameter- free operations such as the max-pool operation without following BN and ReLU do not degrade accuracy. Mathematically, using ReLU after the max-pool operation (without BN) does not give nonlinearity due to the preceding ReI.U. Empirically, involving BN after parameter-free free operation could stabilize the training initially, but model accuracy was improved in the end.

---

Published as a conference paper at ICLR 2022
architectures replacing the spatial operations (i.e., the 333 convolutions) with 1) the I x convolutions; 2) the 3 3 depthwise convolution in entire stages.
Table 2. Model study on ResNet50. We report the performance of kybrid model, which is the most promising model compared with others, and the models with stage-level combinations of the reg- ular bottleneck and the efficient bottleneck (de- further report
Table shows the trade-off between accuracy and speed of the different models mixing the regular and the efficient bottlenecks. B-BB-B B denotes the baseline ResNet50; E B denotes the model using the regular bottleneck and the efficient block alter- nately. Among the different models, we highlight hy- brid and E / B models due to the promising perfor- mance. This shows that only a simple module-based design regime can achieve improved performance over the baseline. E- E-EEEEEEEE the model using the efficient bottlenecks only and shows the fastest speed as we expected. The model with the Ix1 convolution replacements cannot reach the accuracy
noted by and E, respectively). the performance of two variant models by replac- ing each 3x3 convolution with 1) a I 1x] convo- lution; 2) 3x3 depthwise convolution in all the regular bottlenecks. The two trade-off models are in boldface.
Models
GPU Lat. (ms)
Top-1 acc.(%)
Hybrid E/B
7.3
74.9
7.7
75.3
B B +B B
8.7
76.2
E-BB B--B-BB-- B
8.4
75.4
standard presumably due to the lack of the recep- tive field; this is why a sufficient number of spatial operations are required. We observe that the model with the depthwise convolutions shows a promising performance, but our models surpass it as we sim- ilarly observed in $3.3. All the models are trained on ImageNet with the standard 90-epochs training setup" (He et al., 2016a) to report the performance.
E-+E +B BB-BB BB
8.0
75.1
E-EE+H +E
7.3
73.1
E +E-+E>>
6.8
72.0
B- BB-B-BB-B
8.4
75.4
B-BB EE-E
7.7
74.6
B E EE >E
7.3
74.0
1 1 conv 3x3 dwconv
6.2
30.1
8,3
74,9
4.2 EFFICIENT VIT ARCHITECTURES
We provide further use case of the parameter-free operations in totally different architecture from CNN. We again choose the max-pool operation as the parameter-free operation and apply it to replace the self-attention layer (Vaswani et al., 2017) in vision transformer (ViT) (Dosovitskiy al., 2021) to observe how the efficient operation can replace a complicated layer. We do not change the MLP in transformer block but replace the self-attention module with the max-pool operation. Specifically, after the linear projection with the ratio of 3 (i.e., identical to the concatenation of the projection of query, key, and value in the self-attention layer in ViT), the projected dd-features reshaped to 2d-fcatures for the input of the spatial operation. Consequently, the self-attention layer replaced with a cheaper parameter-fie free operation and will bring clear efficiency. We use the global average pooling (GAP) layer instead of using the classification token of ViT since the classification token is hardly available without the self-attention layer. Many transformer-basee baseline; we additionally adopt a strong baseline Pooling-based Vision Transformer (PiT) (Heo et al., 202 lb) to show applicability. More details are elaborated in Appendix A.
ViTs can be a
5 EXPERIMENTS
5.1 IMAGENET CLASSIFICATION
Efficient ResNets. We perform ImageNet (Russakovsky et al., 2015) trainings validate the model performance. We adopt the standard architecture ResNet50 (He et al., 201 as the baseline and train our models with the aforementioned standard 90-epochs training setting fairly compare with the competitors (Luo et al., 2017; Huang & Wang, 2018; Wu et al., 2018; Wang et al., 2018 10; Han et al., 2020; Yu et al., 2019; Qiu et al.. 2021), where the efficient operators were proposed or the networks were pruned. We report each averaged speed of the models with the publicly released codes on a V100 GPU. Table 3 shows our networks acheieve faster inference speeds than those of the com- petitors with the comparable accuracies. The channel-widte pruned models (Slimmable-R550 0.5 x,,, 0.75x) (15xxaaa and the models with new operations (Veratile-RSi) R50, GhostNet-R50, and SlimConv-R500 can-
*Trainings are done with tixed image size 224x224 and the standard data augmentation (Szegedy et al., 2015) with the mondoo_resizedcrrre ccrop rate from 0.08 to 1.0. We use stochastic gradient descent (SGD) with Nesterov momentum (Nesterov, 1983) with momentum of 0.9 and mini-batch size of 256, and learning rate is initially set to 0.4 by the linear scaling rule (Goyal et al., 2017) with step-decay learning rate scheduling; weight decay is set le-4. The accuracy of the baseline ResNet50 has proven the correctness of the setting.
7

---

Published as a conference paper at ICLR 2022
Table ImageNet performance comparison of efficient models. We report the model performance including accuracy, the number parameters, FLOPs, and the GPU latency measured on a V100 GPU. All model speeds are measured by ourselves using the publicly released architectures. used further training recipes.
Network Architecture
Params. (M) FLOPs (G)
GPU (ms)
Top-1 (%)
Top-5 (%)
ResNer50 (R50) (He et al., 2016b)
25.6
4.1
8.7
76.2
93.8
Thinet70-Rae (Luo et al., 2017)
16.9
2.6
72.1
90.3
SSS-R50 (Huang & Wang, 2018)
18.6
2.8
74.2
91.9
Shift-R50 (Wu et al., 2018)
22.x
N/A
75.6
92.8
Versatile-R50 (Wang ct al., 2018)
17.1
1.8
18.7
75.5
92.4
Sllimable- R50 (0.5x) (Yu et al., 2019)
6.9
1.1
8.5
72.1
N/A
Sllimable- R50 (0.75x) (Yu et al., 2019)
14.8
2.4
8.6
74.9
N/A
GhostNet-R50 (Han et al.. 2020)
14.0
2.1
20.3
75.0
92.3
SlmConv-R50 (k=8/3) (Qiu et al,, 2021)
12.1
1.9
24.5
75.5
N/A
Ours-R50 (max)
14.2
2.2
6.8
72.0
90.5
Ours-R50 (hybrid)
17.3
2.6
7.3
74.9
92.2
Ours-R50 (deform_mag) max)
18.0
2.9
10.3
75.3
92.5
Ours- R50--R50 (max)t
14.2
6.8
74.3
92.0
D50 (hybrid)
17.3
2.6
7.3
77.1
93.1
Ours-R50 (eeformmmxx)!
18.0
2.9
10.3
78.3
93.9
Table 4. ImageNet performance of CNN models. We report the ImageNet performance, mCE (ImageNet-C) and AUC (ImageNet-O) of the diverse CNN models. All the redesigned models experience massive reductions of computational costs and substantial gains on mCE and AUC barely drop the accuracy (<2.0%).
Network
Params. (M) FLOPs (G)L GPU (ms))
CPU (ms)
Top-1 (96)
Top-5 (901
mCE (X)
AUC (%of
ResNer50
25.6
8.7
45.4
78.5
94.2
63.8
$1.7
Ours-R50
17.3
2.6 (-37%)
7.3 (-17%)
39.8 (-12%)
77.1 (-1.4)
93.1 (-I.)
$7.5 (-6.3)
52.9 (+1.2)
ResNec50-SE
28.1
4.1 2.6 (-37%)
13.9
98.0
79.5
94.7
69.6
58.9
Ours-R50-SE
19.9 (-29%)
12.5 (-10%)
92.6 (-9%)
78.2 (-1.3)
93.9 (-0.8)
63.8 (-5.8)
59.2 (+0.3)
RetNet101
44.6
7.8
16.7
80.5
80.1
94.9
69.7
$1.7
Ours-R101
26.3 (-41%)
4.3 (-45%)
13.5 (-19%)
65.7 (-18%)
78.2 (-1.9)
93.8 (-1.1)
62.0 (-7.7)
54.6 (+2.9)
WRN50-2
68.9
11.4
9.0
84.0
79.7
94.7
67.0
49.8
Ours- Ours-WRN50-2
36.0 (-48%)
5.4 (-53%)
7.3 (-20%)
59.8 (-29%)
78.1 (-1.6)
93.8 (-0.9)
60.9 (-6.1)
51.9 (+2.1)
WRN101-2
126.9
95.3
73.2
50.6
Ours-WRN101-2
53.9 (-58%)
8.9 (-618)
13.7 (19%)
96.3 (334%)
78.9 (:2.0)
94.2 (-1.1)
63.4 (:9.)
54.8 (+4.2)
not reach the model speed to ours. Additionally, we report the improved model accuracy with further training tricks (see the details in Appendix C). This aims show the maximal capacity of our model even using many parameter-free operations inside. The last rows in the table present our model can follow the baseline accuracy well, and we the gap between the baseline and ours has diminished; this show a potential of using parameter-free operations for a network design.
Bigger Network Architectures. Our design regime is applied to complicated network architec- tures such ResNet50-SE (Hu et al., 2017) and Wide ResNet-101- 101-2 (WRN101-2) (Zagoruyko & Komodakis, 2016). We report the ImageNet performance plus mean Corruption Error (mCF) on ImageNet-C (Hendrycks & Dietterich, 2019) and Area Under the precision-recall Curve (AUC) detection performance on ImageNet-O (Hendrycks et al., 2021).
the measure of out-of--istribun Table 4 indicates the models redesigned with the parameter-free operations work well; bigger mod- els are more significantly compressed in the computational costs. Additionally, we notice all the mCEs and AUC of our models are remarkably improved, overtaking the accuracy degradation, which means the models with parameter-free operations suffer less from overfitting.
Deformable Max-pool Operation. We manifest future direction of utilizing a parameter-fre free operation. We borrow a similar idea the deformable convolution" (Dai al., 2017: Zhou & Feng, 2017). Specifically, we identically involve convolution to interpolate the features predicted by itself to perform a parameter-fres operation on. The max-pool operation still covers the spatial operation; only the offset predictor has weights to predict the locations. We regard this operator to how the performance of the max-pool operation can be stretched when involving few numbers parameters. Only performing computation on predicted locations can improve the accuracy over the vanilla
We implement the operation upon the code: https: /ttps:////tttut httgttttuuub.oo haamm/hhhhreeShhnn//onnnte
8

---

Published as a conference paper at ICLR 2022
Table 5. COCO object detection results. All the models are finetuned train2017/ by ourselves using the
Imaeeeet--retrained backbones Table We report box APs on va12017.
Bbox AP IOU
Backbone
IN Acc. (%)
Input Size
GPU (ms)
Params, (M)
FLOPs (G)
AP
APs0
APps
ResNet50
76.2
x00
32,9
51.8
35.1
42.8
41.8
202.2
ResNet50 (hybrid)
74.9
800
31.9 (-1.0)
51.5
34.1
37,5 (-12%)
33.6 (-20%)
175.8 (-13%)
ResNet50 (deform_max,
75.3
x200xx00
(3.2(+0.3)
53.0
35.3
47.7(-119)
34.3 (-18%)
181.8 (-10%)
Table 6. ImageNet performance of ViTs, We report ViT models' performance trained on ImageNet with diverse training settings denoted by Vanilla, CutMix, and DeiT (with strong augmentations).
Throughput (imgs/sec)
Model
Vanilla
+CutMix
DeiT
256-batch
I-batch
ResNet50
962
112
76.2
77.6
78.8
ViT-S
787
86
73,9
77.0
80.6
VIT-S (dw-conv)
571 (-216)
95 (+9)
76.1 (+2.2)
78.7 (+1.7)
81.2 (+0.6)
VIT-S (max-pool)
763 (-24)
96 (+10)
74.2 (40.3)
77.3 (+0.3)
80.0 (-0.6)
P
952
57
75.5
78.7
81.1
PTT-S (dw-conv)
781 (-171)
90 (+33)
76.1 (+0.5)
78.6 (-0.1)
81.0 (.0.1)
PTT-S (max-pool)
1000 (+48)
92 (+35)
75.7 (+0.2)
78.1 (:0.6)
80.8 (...3)
computation with a few extra costs, as shown in Table 3. Note that there is room for faster operation speed as the implementation can be further optimized.
5.2 COCO OВJECт DETECTION
We verify the transferability of our efficient backbones on the COCO2017 dataset (Lin al., 2014). We adopt the standard Faster RCNN (Ren et al., 2015) with FPN (Lin et al., 2017) without any bells and whistles to finetune the backbones following the original training settings (Ren et al., 2015; Lin et al., 2017) Table shows our models achieve a better trade-off between the AP scores and the computational costs, including the model speed, and does not significantly degrade the AP scores even inside massive parameter-free operations. We further validate the backbone with the deformable max-pool operations in Table 5. Strikingly, AP scores are improved over ResNet50 even with the low ImageNet accuracy; this shows the effectiveness of the operation in a localization task.
5.3 IMAGENET CLASSIFICATION WITH EFFICIENT VISION TRANSFORMERS
We demonstrate using parameter-free operations in ViT (Dosovitskiy et al., 2021) in novel way. We follow the aforementioned architectural modifications. Two vision transformer models ViT- S and PiT-S (Heo et al., 2021b) are trained on ImageNet with three different training settings: Vanilla (Dosovitskiy et al., 2021), with CutMix (Yun et al., 2019) and DeiT (Touvron et al., 2021) settings in Heo et al. (2021b). We similarly apply the depthwise convolution into the self-attention layer. which can be a strong competitor to make a comparison. We report the performance of the models in Table 6; pers (Dosovitskiy et al., 2021; Heo et al., 2021b), which is measured on 256 batch-size and a single batch-size both. The result demonstrates that ViT and PiT with the max-pool operation have faster throughput without significant accuracy degradation compared with the baselines; the depthwise convolution promising alternative, but the throughput is a matter compared with the max-pool operation for both architectures. Interestingly, PiT takes advantage of using the parameter-free op- eration, which presumably comes from larger features in early layers.
report throughput (images/sec) as the speed measure following the ViT pa-
CONCLUSION
In this paper, we rethink parameter-free free operations as the building block of learning spatial infor- mation to explore a novel way of designing network architecture. We have experimentally studied the applicability of the parameter-free operations in network design and rebuild network architec- tures, including convolutional neural networks and vision transformers, towards more efficient ones. Extensive results on a large-scale dataset including ImageNet and COCO with diverse network ar- chitectures have demonstrated the effectiveness case of the parameter- free operations as the main building block. We believe our work highlighted a new design paradigm for future research beyond conventional efficient architecture designs.
the existing efficient architectures and the use

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We would like to thank NAVER AI Lab members for valuable discussions. We also thank Seong Joon Oh, Sangdoo Yun, and Sungeun Hong for peer-reviews. NAVER Smart Machine Learning (NSML) (Kim et al., 2018) has been used for experiments.
ETHICS STATEMENT
This paper studies general topic in computer vision which is designing an efficient network archi- tecture. Therefore, work does not be expected to have any potential negative social impact but would contribute to the computer vision field by providing pretrained models.
REPRODUCIBILITY STATEMENT
We provide detailed information of all the experiments in the paper. Furthermore, the details of our models with specific training syper-paramaters clearly announced for those who would like to design or train our proposed models.
REFERENCES
Irwan Bello, William Fedus, Xianzhi Du, Ekin Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv preprint ar 103::2003.07579, 2021.
Weijie Chen, Di Xie, Yuan Zhang, and Shiliang Pu. All you need is a few shifts: Designing efficient convolutional neural networks for image classification. In CVPR, 2019a.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, 2019b.
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair darts: Eliminating unfair advantages in differentiable architecture search. ECCV, 2020.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical data aug- mentation with no separate search. arXiv preprint 1909. 2019.
2019.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x words: Transformers for image recognition at scale. In ICLR, 2021.
Perry Gibson, José Cano, Jack Turner, Elliot Crowley, Michael O' O'Boyle, and Amos Storkey. Op- timizing grouped convolutions on edge devices. In ASAP, 2020.
Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In ICMI., 2013.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.02677 2017.
Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. In /CLR, 2017.
Dongyoon Han, Sangdoo Yun, Byeongho Heo, and YoungJoon Yoo. Rethinking channel dimensions for efficient model design. In CVPR, 2021.
Kai Han, Yunhe Wang. Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In CVPR, 2020.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016b.
Yihui He, Xianggen Liu, Huasong Zhong, and Yuchun Ma. Addressnet: Shift-based primitives for efficient convolutional neural networks. In WACV, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In ICLR, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
Byeongho Heo, Sanghyuk Chun, Scong Joon Oh, Dongyoon Han, Sangdoo Yun, Gyuwan Kim, Youngjung Uh, and Jung- Woo Ha. Adamp: Slowing down the slowdown for momentum optimiz- scale-invariant weights. In ICLR, 2021a.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In ICCV, 2021
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re- identification. arXiv preprint arXiv:1703.07755 2017.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chied Chich-hhhhhhh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV, 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
Jie Hu, Li Shen, and Gang Sun. gguuezz-andd-exitatiinn networks. In arXiv: /000::/009.0077 2017.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochas- tic depth. In ECCV, 2016.
Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In ECCV, 2018
Sergey loffe and Christian Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. In ICML, 2015,
Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Tong Verma, Vikas ande Che, and Yoshua Bengio. Residual connections encourage iterative inference. In ICIRR 2018.
Hanjoo Kim, Minkyu Kim, Dongioo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. NSML: Meet the MlaaS platform with a real-world case study, arXiv preprint arXiv:1810.09957 2018.
A. Krizhevsky. Learning multiple layers of features from tiny images. In Tech Report, 2009.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. arXiv preprint
2019.
Tsung-X Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In ICCV, 2017.

---

Published as a conference paper at ICLR 2022
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017a.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1.Xiv:1711.05011, 2017b.
Gangzhao Lu, Weizhe Zhang, and Zheng Wang. Optimizing depthwise separable convolution oper- ations on gpus. IEEE Transactions on Parallel and Distributed Systems, 2021.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. ICCV, 2017.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In NeurIPS, 2016.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Y. E. Nesterov. A method for solving the convex programming problem with convergence rate O(1/k²), Dokl. Akad. Nauk SSSR, 269:543-547, 1983.
Jiaxiong Qiu, Cai Chen, Shuaicheng Liu, Heng-Yr Yu Zhang, and Bing Zeng. Slimconv: Reducing channel redundancy in convolutional neural networks by features recombining. IEEE Transac- tions on Image Processing, 30:6434-6445, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause. Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211- 252. 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. CVPR, 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In 2017.
Sihyeon Seong, Yegang Lee, Youngwook Kee, Dongyoon Han, and Junmo Kim. Towards flatter loss surface via nonmonotonic learning rate scheduling. In UAI, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1946, 2019.
Mingxing Tan and Quoe V Le. Efficienttetv2: Smaller models and faster training. ICML, 2021.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-aweraged consis- tency targets improve semi-supervised deep learning results. In NewrIPS, 2017.
12

---

Published as a conference paper at ICLR 2022
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. In NeurIPS, 2016.
Yunhe Wang, Chang Xu, Chunjing Xu, Chao Xu, and Dacheng Tao. Learning versatile filters for efficient convolutional neural networks. NeurIPS, 2018.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative spatial convolutions. In CVPR, 2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In ICLR, 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2 106.020000 2021.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient volutional neural network for mobile devices. In CVPR, 2018.
Pan Zhou and Jiashi Feng. 17Xiv:1705.07036 07038, 2017.
The landscape of deep learning algorithms. arXiv preprint
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In CVPR, 2019.
13

---

Published as a conference paper at ICLR 2022
APPENDIX
A DETAILS OF EFFICIENT ARCHITECTURES
We elaborate on the efficient building blocks used in the experiments above. Fig.A.1 shows the schematic illustration of the proposed blocks compared with the original ones. We observe that the modification simple and readily be applied to any network architecture.
Can 1x
(a) Bottleneck (He al., 2016b)
(b) Transformer (Dosovitskiy al., 2021)
Narm
Eff-layer
Norm
(c) Efficient Bottleneck
(d) Efficient Transformer
Figure A.I. Schematic illustration of the efficient building blocks. We visualize (a) the regular bottleneck
ResNets (He al., 20) (b) the regular transformer in ViTs (Dosovitskiy et al., 2021); (c) our efficient bottleneck; (d) our efficient transformer; the eff-layers in (c) and (d) denote the parameter-fiee operations.
Efficient Bottleneck. We replace the triplet of 3 x3 convolution, BN (loffe Szegedy, 2015), and ReLU (Nair & Hinton, 2010) in the regular bottleneck with single spatial parameter-free operation (denoted as eff-layer Fig.A.Ic). The fastest architecture in Table and Table fully replace regular bottlenecks, including the downsampling blocks with the efficient bottlenecks; therefore, the parameter-fie free operation (here the max muo-p operation) plays role of spatially aggregating the fea- tures by reducing the resolution. The hybrid architecture has almost identical design regime with the most efficient architecture (i.e., E-FE-F in Table 2) except for the downsampling blocks. We remain each downsampling block with the regular bottleneck blocks and involve efficient parameter-free operation before the spatial operation (we assign the avg-pool as the parameter- free operation). We apply the identical bottleneck configuration of the hybrid architecture to the di- verse deep convolutional neural networks (CNNs) in Table 4. Note that we do not modify the depth and the width of network architecture, the stem of CNNs that the first set of layers before the first bottleneck, and the output layer having fully connected layer with 1000-d output dimension. Furthermore, we not modify specific architectural elements such as the SE-block ResNet50- SE (Hu et al., 2017) inside each regular bottleneck but use it in each efficient bottleneck at the same positions.
Efficient Transformer. Designing the efficient vision transformer (VIT) architecture is done by replacing half of the self-attention layers with the spatial parameter-fiee operation (denoted as eff- layer in Fig.A.Id). In other words, the original transformer block and the efficient transformer block are used alternately. We use the 3x3 max-pool operation for eff-layer in the efficient transformer block and compare with the variant using the 5x5 depth-wise convolution for eff-layer. ReLU is added after eff-layer and the first linear layer of the efficient transformer block to give additional non-linearity. For Pooling- based Vision Transformer (PiT) (Heo al., 2021a) and ViT (Dosovitskiy et al., 2021), we do not modify the architectural elements including 1) the patch size; 2) the stem that patchifies the input for the following transformer: 3) the number of transformers. We only change the classification head position from the classification token to the Global Average Pooling (GAP) at the last layer since the classification token not compatible with convolutional operations. As reported in Zhai et al. (2021), transformer with GAP shows comparable performance to ViT with the classification token. We also evaluate a more efficient network architecture which is fully equipped with the efficient transformer; it achieves faster speed, but slightly less accurate. We leave further improvements with a parameter-fi free operation inside the efficient transformer to reach the accuracy of the original transformer future work.
14

---

Published as a conference paper at ICLR 2022
SGD
AdamW
AdamP
2e-3 300
26-3 3.0
1e-3
of paramo
Top-1 acc. (56)
1.2
TP of
155
1.55
1.0
1.0
8.z 0.55
1.20
00
0.50
3.50
2.03
625
350
3,50
2.05 0.5
Channel Exp. Ratio
Channel Exp. Ratio
Channel Exp. Ratio
Figure B.I. Comparison of the parameter-free operations. We visualize top-1 accuracy (solid lines) with accuracy per # parameters (dashed lines) similar to Fig. with two parameter-free operations. All the settings identical the previous study, so only plot the channel width of 32. We observe the max-pool operation consistently heat the avg-pool operation a single bottleneck training.
B ON PARAMETER- EFEEE OPERATIONS
Can We Use Avg-pool as the Spatial Parameter-free Operation? We mainly used the max max-pool pool operation as the spatial parameter-fre operation that replaces the regular convolutions and the self- attention layer experiments. The max-pool operation is expected to have higher expressiveness compared with that of the avg-pool operation. Because the avg-pool operation is a conceptually smoothing operation, and the max-pool operation contains a nonlinearity similar to what ReLU has (i.e., max and max 0)). Experimentally, we found low expressiveness of the avg-pool operation in the shallow network study. Fig.B.1 B.1 shows the comparison of the max-pool and the avg- pool operations trained in single bottleneck block. The setting is identical to the previous study in $3, and we only visualize the case of the channel width of 32 because a similar trend with the channel width of 64 observed. We observe the large accuracy gaps between the two operations. This may be ground of the search results where few avg-pool operations are chosen in normal cells in the previous NAS experiments in $3.3.
Deformable Avg-pool Operation. We here provide an interesting idea of using the avg avg-pool oper- ation, which shows inferior outcomes than the max max-pool operation in the operation expressiveness. We have proposed the deformable max-pool operation involving few parameters to improve the parameter-free operation's discriminative power significantly. Using the same idea into the avg-pool operation is available; surprisingly, a downside of the low expressiveness of the avg-pool operation has been vanished, as shown Table B. This result exhibits that even a smoothing operation can be employed to learn discriminative features in deep neural network.
Table B.I. ImageNet performance of deformable operations. We report the model performance trained on ImageNet for the deformable operations. The new operation, dubbed deformable avg avg-pool, surprisingly reaches the accuracy of the deformable max- pool operation.
Network Architecture
Params. (M) FLOPs (G)
GPU (ms)
Top-1 (%)
Top-5 (%)
Ours-R50 (deform max)
18.0
2.9
10.3
78.3
94.0
Ours-R50 (deform_av)
18.0
2.9
10.3
78.3
93.9
C DETAILED TRAINING RECIPES
We use the following training recipes to maximize the accuracy of the ResNet-based models. We use the cosine learning rate scheduling (Loshchilov & Hutter, 2017a) with the initial learning rate of 0.5 using four V100 GPUs with batch size of 512. Exponential moving average (Tarvainen & Valpola, 2017) over the network weights is used during training. We use the regularization techniques and data augmentations including label smoothing (Szegedy et al., 2016) (0.1), RandAug (Cubuk et al., 2019) (magnitude of 9), Random Erasing (Hermans et al., 2017) with pixels (0.2), lowered weight decay (le-5), and large training epochs (400 epochs) We use the code baseline in the renowned repository for our ImageNet training. The accuracy of the baseline models could reach the known
When training a larger epochs (600 epochs), the top-1 accuracy of Ours-R50 (max) in Table improved to 75.5% which gets closer to the original ResNet's; furthermore, Ours-R50 (hybrid), and Ours-R50 (deform max) reach 78.0 and 79.3, respectively. httpp:: //gi /i thub. com/ rwi wtigggg/mptttrrttthteeee
mage-models
15

---

Published as a conference paper at ICLR 2022
Input
ResNer50
Ours
Figure D.I. Grad-CAM visualization final features. We visualize the highlighted features from output of the final stage using Gradient-weighted Class Activation Mapping (Grad-CAM) (Selvaraju et al., 2017). The first row shows the original images, which are randomly picked from the ImageNet validation set; the second row shows the visualized features of the ImageNet-pretraiing pretrained ResNet50; the last shows the result of our ImageNet-prettiee pretrrindd ResNet50 (hybrid). Ours show similar outputs compared with ResNet50 seem to capture the foreground with wider regions.
improved accuracy reported in such paper (Bello et al., 2021) which presents the training recipes for highly improved models.
D UNDERSTANDING MAX-POOL OPERATION
This section provides intuitive explanations how parameter- free operations such as the max max-pool operation could work as replacing a trainable layers in a network.
D.1 CONNECTION WITH MAXOUT
Maxout (Goodfellow et al., 2013) performs the max operation to set of the inputs like an activation function with the multiple inputs. Maxout is designed use after bunch of linear or convolution layers to ensemble the outputs in a nonlinear manner, which leads to the increase of model capacity. the original paper (Goodfellow
al., 2013) results from the claim that Maxout can approximate any operations, namely perform as a universal approximator. From an architectural point of view, the success of Maxout is probably due to the enhanced model capacity by the increased input dimension with the nonlinearity imposed by the max operation.
The accuracy improvement
For the max-pool operation with the efficient bottleneck mainly used throughout the paper, the re- lationship between the Maxout operation and ours is worth discussing. Maxout outputs through the max operation of the outputs of multiple linear layers computed from a single input. Otherwise, the max-pool operation in the efficient bottleneck performs the max operation with multiple transformed inputs (i.e., transformed pixels spatially adjacent to each point) by the preceding 1xl convolution, which acts single linear layer for given channel. However, if we replace the small variations in neighboring pixels with the perturbations in the weights of the linear layer as claimed in Seong al. (2018), then we may interpret the max-pool operation performs like Maxout.
D.2 RECEPTIVE FIELD MATTER
Since the spatial parameter-free operations have the receptive field (Luo et al., 2016) like convolu- tions, we conjecture such simple operations can capture striking pixels (or patches). We empirically support the conjecture by providing a visualization of the final output features of the ImageNet- pretrained ResNet50 and our ResNet50 (hybrid) in D.1. We observe that two models success- fully localize the foreground object in common for all the images. A noticeable difference that ResNet50 tends to focus more specific crucial regions but ResNet50 (hybrid) on wider regions (a similar trend will be observed in Fig. D.2)..).. Note that the area the highlighted region does not directly link to a model's 's accuracy (in fact, ResNet50 has higher accuracy). We believe that this in- dicates how much a model learns localizable features and can be utilized to understand the learning dynamics of various models.
16

---

Published as a conference paper at ICLR 2022
Input
conv5.2.1
conv5 2.22
conv5_2.
conv5_3.1
conv5_3.
conv5_3_3
R Net50
Ours
RexNer50
Ours
Chrs RosNer50
Figure D.2. Grad-CAM visualization of intermediate features. We visualize the highlighted features hy Grad-CAM of the intermediate features produced by the six successive layers in the last two bottlenecks of the final stage in ResNet50 and ResNet50 (hybrid), respectively. Each visualized feature from left right gets closer to the final output features model. Each feature of conv5 y denotes the output features the y-th layer in the x-th bottlenck,, where a bottleneck has the 1x1 convolutions at the 1st and 3rd layers) and 3x3 convolution/the max-pool operation at the 2nd layer in ResNet50/our ResNet50 (hybrid). The input images are randomly picked from the ImageNet validation set.
We further conjecture that a 1x convolution can complement the expressiveness a parameter-fres operation such the max-pool operation or avg-pool operation by nonlinearly mixing the features, which is computed by the parameter-fie operation. To make a ground for the conjecture, we would like to visualize the intermediate features extracted from 1) each of two 1x1 convolutions; 2) the spatial operations including the x convolution and the max-pool operation in a bottleneck block. We choose the last two bottlenecks in the final stage conv5 and visualize the output features of the aforementioned layers.
D.2. shows the highlighted features produced by the three different input images randomly picked from the ImageNet validation set. We use the identical models employed to visualize the final fea- tures shown in D.1. We let conv5_x.y denote the output features of the y-th layer in the x-th bottleneck of a specific model; for example, each visualized feature of conv 5_2.2 in ResNet50 and our model (ResNet50 (hybrid)) indicates the output features of the 3x3 convolution and the spatial max-pool operation, respectively. First, we observe the 1 xx convolutions refine the features to be more discriminative; when comparing the features in conv5 x__ with those of conv5_x_3, the output features usually get more highlighted on the crucial region in the foreground objects. Furthermore, seems that the 1x convolutions change more the previous features in our model, which may come from the different output features by the 3x convolutions and the max-pool operations (com- pare the output features of ResNet50 with ours in conv5_x.2). Therefore, based on the observations, the 1x1 1 convolution performs to make features more discriminative, and they do more with the parameter-free operations like the max-pool operation.
E MORE VISUALIZATIONS
In this section, we further discuss the studied materials with the additional graphs and figures to provide more detailed information. $E.1 contains extended trade-off graphs of our study in $3.3;
17

---

Published as a conference paper at ICLR 2022
22
52
92
1.000
68
2.6
2.8
3.5
3.2
17.5
10.0
38.5 38.0
18.5
0.32
0.14
0.15 0.18
2200
5.22
25.
25
(a) Speed (1-image) (b) Speed 256-images
(c) # parameters
(d) FLOPs
Figure F.I. Multiple bottlenecks study (cont'd). The entire models trained the multiple bottlenecks study in $3.3 are visualized in the comparison graphs: (a) Accuracy vs. speed with the batch size of 1: (b) Accuracy vs. speed with the batch size of 256; (c) Accuracy # parameters; (d) Accuracy vs. FLOPs. The graphs includes the 20% best-performing models plotted Fig.2. The depthwise convolution (blue dots) shows efficiency again # parameters and FLOPs, but the max-pon pax-pool (i.c., red dots) clear benefits the speed measures. Note that the speed gap between the two operations gets larger when processing multiple images, as shown through (a) and (b).
Deni
Pean
Figure E.2. Visualization of predicted offsets. We illustrate the predicted offsets deformable max-pool operation from images the validation set in ImageNet. Each sampled image two predicted offsets on foreground object (left) and the background (right). We plot 1) the predicted offsets (red dots); 2) the initial grid-like points (blue dots) with the center (green dots) in each figure.
$E.2 discusses predicted offsets by the proposed deformable max-pool operation; finally in $E.3, discuss the performance trade-offs of the models on the ImageNet datasets $5.1.
E.1 MULTIPLE BOTTLENECKS STUDY (CONT'D) D)
Fig.E.1 shows the performance of the entire models which are trained in $3.3. The graphs includes the 20% best-performing models shown in Fig.2. As expected, the models using many max- poxo operations have degraded accuracies but show clear speed benefits; there is about 1.5% accuracy gap between the fastest models using each operation yet more than a 5ms speed gap between them.
E.2 PREDICTED OFFSETS BY DEFORMABLE MAX-POOL OPERATION
We visualize the predicted offsets by the proposed deformable max-pool operation (deform max) with the sample images in F.2. We plot the aggregated offsets of the last deform maxs in the final stage ResNet50 with the initial points, which look like the grid-like offset points of the two successive regular convolutions. We observe the offsets are concentrated on the objects when the center is on each foreground object. The offsets spread widely when the center is on the background, which is similarly observed in the deformable convolution paper (Dai al., 2017: Zhu et al., 2019). It is surprising that our models only trained with the ImageNet's class labels without strong supervisions (i.e.. detection boxes or segmentation masks) show the improved localization capability. Furthermore, albeit our model does not be trained with the background class, each shape of the predicted offsets on foreground and background looks quite different as shown in Fig.
We would like to stress the main differences of the experimental settings with those in the deformable convolution papers. First, our model trained on ImageNet only with class labels without any strong supervision so that the models may have weaker localization capability than the models trained with more supervision. such as on detection and segmentation tasks. Second, our model incorporates much more deformable operations in the entire stages of ResNet. the dynamics of predicting offsets would be different from the deformable convolution where the only later stages have the deformable convolutions.
18

---

Published as a conference paper at ICLR 2022
E.3 PERFORMANCE TRADE-OFFS
This work presents the potential usefulness of parameter-free operations such as the max-pool as building block rather than pushing the performance to the limit. We first visualize the ImageNet performance comparison with many efficient models shown in Table in E.3. Fig.E.3 shows our models have competitive trade-offs between accuracy and speed. Since we do not modify ResNet50 but just replace the spatial operation into parameter-free operations such as the max- max-pool, our models do not have much benefits in the number of parameters and FLOOPs
Moreover, we visualize the ImageNet results including 1) top-1 accuracy on ImageNet; 2) mean Cor- ruption Error (mCE) on ImageNet-C: 3) Area Under the precision-recal Curve (AUC) on ImageNet- O applying the efficient bottleneck into existing big CNNs, which is shown in Table 4, in Fig. E.4. Fig.E.4 shows our models achieve large improvements on the entire computational costs but show similar trade-offs of the baseline models in top-1 accuracy. This experiment originally aims to in- vestigate the redundancy of using standard building blocks inside heavy CNN models. However, a simple replacement of the operations in existing models achieves significant efficiency; namely, in terms of the model speeds, the number of parameters, and FLOPs, our models reduce meaningful amounts without much accuracy loss. Furthermore, our models significantly outperform the baseline models with efficiency in mCE and AUC measures, Improving the ImageNet top-1 accuracy while having the advantages of the parameters- operations will be our future work.
80
00
80
75
75
Sapur
Swer
Ours
Ours
Diter models
models
Other models
GPU Spead
25
15
(M)
FLOPs (G)
(a) GPU speed
(b) # parameters
(c) FLOPs
Figure E.3. Performance Trade-offs of the models in Table 3. We visualize the trade-offs between (a) accu- racy and speed; (b) accuracy and # parameters; (c) accuracy and FLOPSs respectively. Ours denote the perfor- mance of Ours-R50 (max), Ours-R50 (hybrid), and Ours-R50 (deform.max); we also plot the models trained with the further training recipes.
80
Top-1 acc. (96)
80
Top-1 acc. (99)
80
T0p-1 acc.
60
60
Basaline Models
Bazeine Modal
Bascline Models
14
16
100
125
10
20
GPU Speed (ms)
of params. (M)
FLOPs (G)
70
70
MCE (96)
mCt (96)
MCE
60
60
50
Bassline Models
Durs
50
Baseina Medds
Ours
50
Baseline Modals
Ours
10
12
14
16
25
50
75
100
125
10
20
GPU Speed (ms)
* of params. (M)
FLOPs (G)
60
AUC (%6
AUC
50
AUC
50
Ours
Ours
Basellne Models
Basaline Models
Buseline Models
16
75
100
125
15
GPU Spead (ms)
(M)
FLOPs
(a) GPU speed
(b) # parameters
(c) FLOPs
Figure E.4. Performance Trade-offs of the models in Table 4. We visualize the trade-offs between accu-
racy/error and (a) speed; (b) # parameters; (c) FLOPs, respectively. Each row employs 1) Top-1 accuracy on ImageNet; 2) mean Corruttion Error (mCF) on ImageNet-C; 3) Area Under the precision-recall Curve (AUC)
on ImageNet-O O the performance measure, respectively.
19


Reviews:
[{"review_id": "Y_LEgJ6TYvu", "reviewer": "Reviewer_KvAV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The framing of the method as more general parameter-free operation is over-stated for what has been demonstrated, which is limited to max-pooling (and average pooling in a small number of cases).  In particular, the abstract mentions \"parameter-free\" several times, and \"max-pooling\" not once.  (A second operation, deformable max-pool, is mentioned, but only as a direction for future work, and is not studied beyond one small experiment in the appendix).  Likewise, calling the new layers \"efficient layers\" seems a little inflated to me, if they are just max-pooling substitutions.  It seems more accurate to call them \"max-pooling bottlenecks\" or something specific to max-pooling.\n\nStill, the empirical results are intriguing, showing that simple max-pooling ops can be used in place of convolutions much more than I might have expected.\n\nAlthough I feel I was able to follow most of the paper well enough to understand the technique, I did feel like I was guessing a bit at several key points, and would have liked to have seen more precise definitions.  I have listed these in below detailed comments.\n\nOverall, the empirical results are quite interesting, showing that max-pooling can be used in place of convs or depth-wise convs more extensively than I might have expected, and offers a meaningful datapoint in the wider context parameter-free operations.  While the most specific claims are supported by the experiments, I think the overall framing as a demonstration of parameter-free layers is over-generalized for what has been demonstrated around max-pooling.  There are also many details that I was unclear on, and feel could be more clearly described (below).\n\n\nAdditional comments/questions:\n\n\n* A couple older works might also be interesting to discuss\n  - Maxout Networks, Goodfellow et al 2013 https://arxiv.org/pdf/1302.4389.pdf --- uses max as a nonlinearity in place of relu\n  - Inception v1, v2, v3, ... etc networks also use max pooling as a branch, and some of the cells found in the NAS in this paper look similar to Inception branches\n\n* sec 3.2:  The use of W = s(g) doesn't quite correspond to what I think is the intended operation.  \"W_v,:,: = s(g)\" looks like there is one W matrix that is constructed globally based on the entire feature map g, and used used as a convolution kernel.  Instead, I think the intent is to describe a per-window operation, in which case there should be a dependency on the window location i,j.\n\n* sec 3.2:  i,j are used twice in W, argmax description:  once in the definition (i,j)=argmax_hw, and once again in the expression in the argmax g_v,r*i+h,r*j+w.  most likely the i,j inside and outside the argmax are different (new names would help here).\n\n\n* sec 3.3 multiple bottlenecks experiment:  I had trouble following the description of this experiment.  What was the exact architecture, and how were the replacements of convolutions performed?  How many layers were replaced at a time, and with which operations?  My best guess here, is that starting from a resnet with convolution residuals, new models were created by a procedure where (1) a set of one or more layers were selected for operation replacement (how?), and then (2) in these layers, 3x3 convolutions were replaced with either (a) max pooling or (b) depthwise convolution, creating pairs of models for cases (a) and (b).  The best 20% of models in each (a) and (b) conditions are then plotted in red and blue in Fig 2.  Is this correct?\n\n* Fig 2:  It would also be interesting to see not just the best 20%, but also the other 80%, what the performance looks like as it degrades.\n\n\n* sec 4.1:  This section is the first use of the phrase \"efficient bottleneck\" which is a little confusing, as it would seem the same module was investigated in the cifar10 experiments as well, and described in sec 3.  I think it would be clearer to define the \"efficient bottleneck\" in sec 3 using an equation that closely parallels eqns (1)..(4).  I believe that the \"efficient bottleneck\" operation is the eq (2) but replacing the innermost double-sum (over space window and \\roh*c channels) with per-channel max pooling over k sized windows.\n\n\n* transformer and Fig A.1:  I'm not sure exactly where the maxpooling operation is applied.  In fig A.1(b), there are three paths into the attention layer, which seem to indicate these correspond to q,k,v.  But the same three paths go into the eff-layer in A.1(d) --- does this layer actually combine these three sets of features somehow, or is it applied to just one set?\n\n\n* \"deformable max-pool operation\":  how are the predicted locations are found for this operation, are there parameters involved in this step?  what parts do or do not involve parameters?\n\n\n* Tables 3,4:  It would be informative to plot the first three columns of tables 3 and 4 vs top1 in a set of scatterplots, to better see the speed/performance tradeoff and whether this method moves along the frontier or pushes it.\n\n* Fig 1:  would be nice to have the condition names in left/top of the plot grid, not just the caption\n\n", "summary_of_the_paper": "This paper argues for more extensive use of max-pooling layers in image classification networks, as an inexpensive substitute for convolutions.  Recent works have removed parameters (and often-expensive operations) using bottlenecks, depthwise/channel-wise operations, etc., and this paper pushes further in this direction by studying max-pooling as a cheap and effective spatial combination mechanism in the context of recent resnet-derived and image transformer architectures.  Empirical experiments are performed on CIFAR-10 and three Imagenet variants, demonstrating the effectiveness of incorporating max-pooling layers beyond applications in strided layers and secondary branches.\n", "summary_of_the_review": "Overall, the empirical results are quite interesting, showing that max-pooling can be used in place of convs or depth-wise convs more extensively than I might have expected, and offers a meaningful datapoint in the wider context parameter-free operations.  While the most specific claims are supported by the experiments, I think the overall framing as a demonstration of parameter-free layers is over-generalized for what has been demonstrated around max-pooling.  There are also many details that I was unclear on, and feel could be more clearly described.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ORQjDdbfIBM", "reviewer": "Reviewer_PJUA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n1. I think the idea of replacing the spatial operation (e.g., 3x3 conv) with some parameter-free/cheap operations is good. And the results showed that even simple max pooling could work well, which is inspiring.\n2. The author conducted thorough ablation studies and evaluations on different architectures and datasets. These experiments clearly demonstrated the advantages and disadvantages of the method.\n3. The paper is well written.\n\nWeakness:\n1. I think replacing the spatial operations with some cheap operations is a good idea, but the author could dive deeper into this. Currently, the author mainly conducted this with max pooling, and it achieves reasonable results but not surprisingly good. For example, in most cases, the inference speed is improved by 10%-20% but the accuracy is also decreased by 1-2 points. This seems to be a pretty normal accuracy-efficiency trade-off. \n2. The author tried some improvements, i.e., deform_max, which I think is a good direction. But there are few experiments on it (only in Table 3). I am curious about how the deform_max works without improved training tricks and on other tasks.\n3. The author claimed that depthwise conv is light in params/flops but heavy in inference speed (which I agree with), while the built-in max pooling is highly optimized and efficient. But in Table 4, the flops seems to be reduced by 30%+ while the inference time is only reduced by 10%-20%. Could the author explain this?\n4. The author claimed the deform_max could be optimized further to speed up. I think the author could work on that and show how much speedup it has.\n", "summary_of_the_paper": "This paper proposed to use some parameter-free operations to replace the spatial operations (e.g., 3x3 conv, 3x3 depth conv). The author shows that using the parameter-free operations (e.g., max pooling) can also achieve a good performance. The author conducted thorough ablations to show the effectiveness of parameter-free operations, and the method is evaluated on different architectures and datasets.", "summary_of_the_review": "Overall, I think this paper proposed an inspiring idea but the author could explore further on this. The current results are not surprisingly good but promising. It would be good If the author could further improve the method.\n\n=============== Post rebuttal ================\n\nThank the authors for the detailed response. I am satisfied with most of it. I still have some concerns about the proposed deform_max and its performance. The idea of replacing conv with pooling (or general parameter-free operations) seems promising. But the results in the paper are not suprising yet, even with the deform_max. The accuracy is about 1% lower than baseline, with 1.2 GFLOPs less computation but even more inference time. It would be good if the author could further improve and optimize their method. I would like to keep my score as weak accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Cvt-AD_lEy4", "reviewer": "Reviewer_DuSH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ A simple yet effective finding of using parameter-free operations to achieve speedup with reasonable performance drop.  \n+ Very extensive experiments to support authors' claim on the effectiveness of the parameter-free operations.  \n\nCons:\n- In spite of extensive experiments to demonstrate the effectiveness, it lacks some insightful explanation on the reason why it works, why it could achieve a good balance between speed and performance.\n- In the experiment of hybrid architecture with efficient bottlenecks, what about putting the designed efficient block at different position of the architecture? What about alternatively stack baseline and efficient blocks?\n- The performance drops more 1% in AP while reducing less than 5 ms running time in the object detection task. It seems not to be a good tradeoff between performance and speed. Any explanation on such performance in the object detection task? Any solution to address this issue?\n- Why does ViT-S with maxpooling shows slight improvement in performance while drop in speed?", "summary_of_the_paper": "In this paper authors propose to use simple parameter-free operations to replace several kinds of trainable layers to achieve speedup on hardware with less performance drop. Extensive experiments have been done to demonstrate the effectiveness of the proposed method, including experiments on replacement in both single and multiple bottlenecks and neural architecture search. In addition, authors also employ such parameter-free operations to redesign CNNs and vision transformers for a good balance between speed and performance.", "summary_of_the_review": "This paper shows a simple yet effective finding and validate this with extensive experiments. However, it lacks some insightful explanation and novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7oByPrAse9-", "reviewer": "Reviewer_HgeU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Simple but novel idea. The first attempt to investigate built-in parameter-free layer as a building block for network efficiency.\n2. Good and convincing empirical studies. The proposed method achieves compelling results on ImageNet dataset (Table 3, best trade-off for GPU), with detailed analysis on different network variants (CNN and ViT, + deformable operations etc).\n3. Clear and sound technical presentation. The paper clearly elaborates the technical design and insight, and illustrates the difference with existing efficient building block. \n4. Interesting and inspiring findings. For example, similar performances have been achieved using max-pool and avg-pool operation (Appedix B).\n\nQuestions:\n1. Is there any other parameter-free operations that could possibly achieve good trade-off except max and avg operation? \n", "summary_of_the_paper": "This paper focuses on designing efficient deep networks with a proposed novel parameter-free layers. Drawing the spirit of depth-wise convolution, shift operation, the paper studies the ways of inserting parameter-free operations into the building block using NAS technique. The resulted models perform both effectively and efficiently on ImageNet dataset.", "summary_of_the_review": "A good paper. I definitely suggest acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the extent to which parameterized layers within a CNN can be replaced by parameter-free layers, with specific focus on utilizing max-pooling as a building block.  After the author response and discussion, all reviewers favor accepting the paper.  The AC agrees that its empirical results open a potentially interesting discussion on network design.
Decision: Accept (Poster)