FACS: FAST ADAPTIVE CHANNEL SQUEEZING
Authors: Ashish Kumar
Laxmidhar Behera
Venue: Submitted to ICLR 2023 (2023)
Keywords: Fast Channel squeezing
Edge Devices
CNN

Abstract: Channel squeezing is one of the central operations performed in CNN bottlenecks to reduce the number of channels in a feature map. This operation is carried out by using a 1 × 1 pointwise convolution which constitutes a significant amount of computations and parameters in a given network. ResNet-50 for instance, consists of 16 such layers which form 33% of total layers and 25% (1.05B/4.12B) of total FLOPs or computations. In the light of their predominance, we propose a novel “Fast Adaptive Channel Squeezing” module which carries out the squeezing operation in a computationally efficient manner. The key benefit of FACS is that it neither alters the number of parameters nor affects the accuracy of a given network. When plugged into diverse CNNs architectures, namely ResNet, VGG, and MobileNet-v2, FACS achieves state-of-the-art performance on ImageNet and CIFAR datasets at dramatically reduced FLOPs. FACS also cuts the training time significantly, and lowers the latency which is particularly advantageous for fast inference on edge devices. The source-code will be made publicly available.

Under review as a conference paper at ICLR 2023
FACS: FAST ADAPTIVE CHANNEL SQUEEZING
Anonymous authors
Paper under double-blind review
ABSTRACT
Channel squeezing is one of the central operations performed in CNN bottlenecks reduce the number of channels in feature map. This operation is carried out by using a 1 x pointwise convolution which constitutes a significant amount of computations and parameters given network. ResNet-50 for instance, consists of 16 such layers which form 33% of total layers and 25% (1.05B/4.12) of total FLOPs or computations. In the light of their predominance, we propose a novel "Fast Adaptive Channel Squeezing" module which carries out the squeezing operation a computationally efficient manner. The key benefit of FACS is that it neither alters the number of parameters nor affects the accuracy of a given network. When plugged into diverse CNNs architectures, namely ResNet, VGG, and MobileNet-v2. FACS achieves state-of-the-an performance on ImageNet and CIFAR datasets at dramatically reduced FLOPs. FACS also cuts the training time significantly, and lowers the latency which is particularly advantageous for fast inference on edge devices. The source-code will be made publicly available.
INTRODUCTION
Introduced by ResNet (He et al., 2016), squeeze-and-expane units form the basis of taate-of-the-ard CNNs. It is essentially a three layered unit in which the first layer (1 x 1) performs channel squeezing while the third layer (1 1) performs channel expansion. The middle layer (3 x 3), on the other hand maintains the channel count, and governs the network's 's receptive field. Interestingly, in CNNs inheriting suueeze-and-eppaae
units, we make key structural observation; that the squeeze and expand layers dominate both in the number and computations, while do not contribute to network's s receptive field due their pointwise nature. For instance, ResNet-50 consists of 32 such layers out of 50 total layers, accounting for ~ 54% (2.23B 12B) of overall FLOPs, whereas ResNet- consists of 66 of them out of 101, accounting for 50% (3.98B/ 77.BB) of total FLOPs.
As CNNs widely used in machine vision (Ren al., 2015; Zhao al., 2017; Carion al., 2020), bigger networks are now preferred to achieve higher accuracy Gao et al. (2018) to deal with increased task complexity. For this reason, VGG, ResNet style networks are still dominant in both academia and industry (Kumar & Behera, 2019; Ding et al., 2021; Kumar et al., 2020) due to their architectural simplicity, customizability, and high representation power, in contrast to newer complex networks (Tan & Le, 2019). However, as ResNet-like CNNs are based on squeeze-expand units, our preceding observation raises an important question: "can computations without sacrificing network parameters and accuracy?" If so, the inference of such networks can be significantly accelerated on edge computing devices, benefiting a whole spectrum of applications such autonomous driving, autonomous robotics, and on.
x layers be reduced
To the best of our knowledge, the above problem has not been addressed previously but is of great practical importance. Hence, in this paper, we show that indeed it is possible achieve the desired objective by examining channel squeezing through the lens of computational complexity. To this end, we propose novel "fast adaptive channel squeezing" (FACS) module that transforms a feature map X E #C into another map Y E R HW mimicking the functionality of a squeeze layer but with fewer computations, while retaining network's parameters, accuracy, and non-linearity.
We evaluate FACS by embedding it into three CNN architectures: plain (VGG), residual (ResNet). and mobile-series (MobileNet-v2), and on three datasets: ImageNet, (Deng et al., 2009), and CIFAR- 10, 100. FACS is backed with comprehensive ablation study. Since FACS is novel, we demonstrate intermediate data representations learnt by FACS using GradCAM (Selvaraju et al., 2017), and show that they are better relative to the baselines. FACS brings huge improvements e.g. ResNet-50 becomes ~ 23% faster with 0.47% improved accuracy on ImageNet (Deng et al., 2009), whereas VGG becomes faster by 6% with 0.23% improved accuracy, without bells and whistles.
Next section deals with the related work. Sec. 3 describes the FACS module, and its integration into different CNNs. Sec, carries solid experiments in support FACS, Sec. 5, concludes our findings.

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
2.1 CONVNETS
The earlier CNNs (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016) accuracy oriented but have grown complex in terms of branching (Huang et al., 2017), despite having higher accuracy. Mobile networks (Sandler al., 2018; Zhang et al., 2018) focus lower FLOPs for high speed inference by using depthwise convolutions (Sifre & Mallat). However they suffer from low representation power, need prolonged training schedule of 200 400 epochs in contrast earlier ones which are typically trained for 90 120 epochs. The limited representation power hinders their performance on downstream tasks e.g. (Howard et al., 2017) needs 200 epochs on ImageNet to perform similar to (Simonyan & Zisserman, 2014) (trained for 75 epochs), but performs poorly on object detection. (Tan & Le, 2019) tackles this issue by being efficient in parameters and FLOPs, however ends-up being highly branched and deeper, memory hungry, and sluggish (Ding et al., 2021). Recent Transformer based methods are computationally hungry due to their attention mechanisms (Vaswani et al., 2017; Dosovitskiy al., 2020) which put them out of the reach of edge-devices.
ResNet design space exploration (Schneider et al., 2017) provides several variants competitive to (Tan & Le, 2019) like designs while being simpler, quite advantageous for edge devices and real-time applications. Similarly, (Ding et al., 2021) improves the speeds of VGG however its training time design has many branches which is still inferior to ResNet. Furthermore, (Hu al., 2018; Woo et al., 2018) propose novel units which yield improved accuracy in ResNet at the expense of increased parameters because of additional convolutions, and marginal computational overhead. The above discussion shows that the older architectures still have a room for improvement, and real-world applications can be benefited by focusing on these areas.
2.2 FAST INFERENCE
Fast inference of CNNs has been widely explored which mainly includes static pruming, network compression / quantization. These methods generally employed post training and are agnostic to any given network. More recently, dynamic pruning methods such as (Gao et al., 2018) have become state-of-the-art in this area, which deals with the limitation of static pruning methods. Pruning works mainly by disabling or suppressing a set of channels in a feature maps, and for which convolution computations are inhibited. For example, (Gao et al., 2018) consumes tensor X Hx and outputs another tensor Y E xC**#XW but some of the channels in Y have zero values for which convolution operations are inhibited, thus saving computations but at the cost of accuracy. (Gao et al., 2018) achieves its goal by introducing new convolution layers into network.
The proposed FACS orthogonal the above methods since it performs computationally efficient channel squeezing, transforming a tensor X E C to Y E R x In addition, FACS preserves a network's parameters and accuracy, and does not involve new convolution layers, rather reuses the parameters of the original squeeze layer while redefining the information flow. Hence, FACS and pruning can not complement each other. However as pruming is applicable to any network architecture, and as FACS is an architectural enhancement, thus it can be seamlessly integrated with any pruning method to speed-up the overall network. However, in this paper, have limited ourself to the study of FACS in deep networks.
3 FACS
Given an input feature map X E RC FACS progressively infers intermediate 1D descriptors E CxIX1 P R xIx1 and an output feature map Y R x of reduced dimensionality. FACS can primarily be sectioned into three stages: First, global context aggregation which provides channel-wise global spatial context in form of 1D descriptor z. Second, cross channel information blending which transforms z into another descriptor p. referred as adaptive channel fusion probability. Third, channel fusion which utilizes p and X in order to produce Y. The overall structure of FACS module is illustrated in Figure 2.
For reference, we have also shown the baseline channel squeezing method in Figure 1. It can be noticed that FACS is substantially different from the baseline counterpart in term of structure. The only component shared between them is the convolution layer, yet the tensors on which it operates are different. Further, FACS commonly employs global pooling, maximum or averaging operations which are the most fundamental operation in neural networks, however their use in FACS configure infigurationn is entirely novel. For this reason, we term FACS as novel module to perform fast channel squeezing.
2

---

Under review as a conference paper at ICLR 2023
X S
Ye TERS*BON
Coas 1x 1. L.W 2
Figure 1: Baseline squeezing.
c R.xMa
ER
Y- BR
Globul Pooling
Conv x W ER
Charne Fasins
Figure 2: FACS block. of-) denotes sigmoid non-linearity.
3.1 GLOBAL CONTEXT AGGREGATION
The first and foremost step of FACS is to gather global spatial context. For this purpose, we employ global average pooling over the per-channel activations of the input feature map X. We apply the operator on X which results in channel-wise global contextual information in form of a 1D descriptor z e [xxx1 In the computations, each element 24 corresponding 7th channel of is given by:
2,
1
Σ X, w)
(1)
H X W
AEN
The use of global pooling in our case is inspired by its success to aggregate global statistics in a computationally efficient manner e.g. the end of the classification networks (He al., 2016; Hu et al., 80;8; Woo et al., 2018)
3.2 CROSS CHANNEL INFORMATION BLENDING
From the previous step. we obtain global context which limited a single channel only, therefore the descriptor is devoid of cross-channel information. To mitigate the issue, the per-channel global information in z is now blended with the other channels by using a 1 convolution operator. In addition to the blending, the operator also reduces the input number of channels by factor of R i.c. number of input channels are reduced from C to C/R. It is important to note that the convolution operator is same the squeeze layer in Figure 1, but operates only upon a 1D descriptor instead of 3D tensor. This is where the majority of computation savings occur in FACS relative to the baseline.
Next, the output of the convolution operator is squeezed into range E [0, 1] using sigmoidal activation, resulting in 1D descriptor p RC/R, We refer the descriptor p as adaptive channel fusion probability which accommodates both the spatial, and the channel context. Now both the and X are forwarded to the channel fusion stage which yield the final output Y of the FACS module.
3.3 CHANNEL FUSION
In this stage, the task of channel squeezing of X is performed in an adaptive and computationally efficient manner. We define two reduction operations: maximum (Max), and average (Avg). During fusion, the input feature map is first virtually partitioned into chunks of R. channels, and then the channels of each chunk are adaptively fused to produce a single channel. In other words, fusion operation results in C/R output channels, if the number of input channels were C.. We devise two strategies to perform this task, as discussed below.
3.3. THRESHOLD BASED FUSION
In this strategy, the channels are fused by applying the reduction operator (Max or Avg) at each of the spatial location E H x W across R channels. For any spatial location (h, w) E H x W, there exist R scalar values which are reduced The selection of the reduction operator is entirely adaptive because it is determined by the fusion probability p, which turn learnt by the network itself during the training. In short, as soon as the reduction operator is selected, the values at any given location are fused via the following equations:
single scalar by using one of the reduction operator.
Pss Max X, P: Avg(X,(h, x)
w)); < Tef
Y, w)
Pe > Tef
where, Ps is the th element of p. and Tef is channel fusion threshold. This is a hyperparameter which is set to 0.5 based on our ablation study (See Sec. 4). Y,(h, w) is the value at the (h, w) spatial location of the 7t channel in the output feature map Y. i E [0, C/R] and j = R * i. A pictorial represntation of this fusion strategy is provided in Figure 3a.
3

---

Under review as a conference paper at ICLR 2023
B Tol.
Max
Tu
E RERORW
Max
e RRXKoW
Avg
(a)
(b)
Figure 3: Channel Fusion strategies. (a) Threshold based fusion, and (b) weighted fusion.
3.3.2 WEIGHTED FUSION
In contrast to the threshold based fusion, this strategy is free of Taf. In this strategy, for any spatial location (h, w) E H x W, all the R scalar values are first reduced by using both of the reduction
operators separately, and then the resulting values are combined using a weighted sum as given below.
Yi(h, w) p4 * Max (5,h,, w)...Xj++ (h, w)) + (1 - (1-))) * Avg (X,( w), (h, w))
(3)
In this strategy, the network itself learns to determine how much contribution should be given to a reduction operator. Due to this flexibility, each location in chunk can enjoy different combinations of the reduction operators. This is not the case with threshold based fusion. Infact, threshold based fusion becomes special case of weighted fusion when pt approaches the two extremas i.e. {0, 1}.
Figure 3b illustrates the weighted fusion strategy.
In both of the above techniques, the use of p helps choosing or weighting different reduction operators adaptively based on the input. Without that, all the squeeze layers shall have be eliminated which will reduce to the case of (Hussain & Hesheng, 2019), leading to a heavy accuracy loss (Table 2).
3.4 A THEORETICAL VIEWPOINT
Theoretically, FACS does not introduce any parameter overhead instead reuse the parameters of the squeeze layer. As squeeze layer has learnable parameters, therefore it also contributes in the non-linearity of the network. Hence, FACS deals with two problems. First, preserve non-linearity, and Second, reduce computations. The computation reduction primarily occurs due to collapsing the input tensor X E CR* into a 1D descriptor z E RCC Global pooling can potentially introduce information loss, therefore loss of non-lineann non-linearity. However, the proposed learnable cross channel information blending p and the fusion operator works in such a way that non- non-lincarity is maintained.
In this process, role of p is crucial which is used for aggregating knowledge across chunk of R channels at a given spatial location in a feature map to dynamically decide the choice of fusion operator. In case of max fusion, only one channel out of goes out while the AVG pooling, the output have an effect of all the channels in the chunk. Interestingly, each chunk of R channels can dynamically undergo either Max or Avg fusion, depending on p. Hence, each spatial location has flexibility to choose among Max or AVG. For this reason, at any given spatial location E H any one the R channels make it to the output max pooling, while at another location, some other channel wins. This degree of freedom introduces a high level of non-linearity into the network which helps achieving FACS slightly better accuracy even at significantly lower FLOP requirements. These improvements brings huge training and inference speed-ups and are empirically verified in the Sec. 4. Effect of FACS on FLOPs and memory is discussed in appendix 5.
3.5 FACS INSTANTIATION
NesSeR ResNet is enhanced with FACS by replacing all the residual squeeze-expand units (Figure 4a) with FACS ones (Figure 4b). We experiment with ResNet-50, ResNet-101, and ResNet- 152, and add "FACS" prefix the resulting networks.
FGGS-VGG. Although VGG (Simonyan & Zisserman, 2014) does not have a provision of squeeze layers since consists only of 3 x 3 convolution (Figure 4c), we also pave way to enhance them with FACS, due to its popularity in robotics and other real-time applications. To achieve that, we insert 1 convolution before each 3 x 3 convolution layer except the first (Figure 4d)). The purpose of this layer is to function as a squeeze layer which reduces the input channels by a factor of R. We name the resulting model as VGG-1 1 which is enhanced with FACS as shown in Figure 4e.
MACllMobileNNetvv. We also study FACS for mobile networks MobileNet-v2 (Sandler al., 2018) which employs inverted residual units with depthwise separable convolutions. The baseline inverted residual unit and its FACS enhanced counterpart is illustrated in Figure 4f, 4g.
4

---

Under review as a conference paper at ICLR 2023
To
Cres W gooerir
Caw.W
Coen gratissee
FACS
Cres gPPriSsSE
(a) ResNel
(b) FACS RecNet
To goxka Tea
Tegi
(c) VGG
(d) VGG-1 x
(c) FACS-VGG-1 x
Coer,
Crec www.eooooeoooee.
(Car W
Can,
(1) MobileNet-22
(g) FACS--obllNeew''
Figure Zoom in. (a) Squeeze expand residual unit, (b) FACS variant, (c) VGG block, (d) VGG-1x1 block, (e) FACS variant, (f) inverted squeeze-expand unit, and (b) FACS variant.
EXPERIMENTS
We evaluate FACS on ILSVRC'2012 ImageNet (Deng ct al., 2009) having 1.28M training and 50K validation images over 1000 categories. For illustration of transfer learning, use CIFAR-10 and CIFAR-100 image classification datasets. We also evaluate FACS onto a downstream task of semantic segmentation against popular pipeline (Zhao et al., 2017). We first show comprehensive ablation study of FACS design, and then examine impact of FACS into previously instantiated networks. We also provide crucial throughput and latency improvements over five different GPU hardware including edge devices. All the baselines are retrained in PyTorch (Paszke et al., 2019).
Training Specifications. The training procedure kept straightforward to ensure reproducibility. We use a batch size of 256 which is split across 8 GPUs. We use a RandomResized crop of 224 x 224 along with horizontal flip. We use SGD with Nesterov momentum of 0.9, base_ base_1r=0.1 with CosineAnnealing (Loshchilov & Hutter, 2016) learning rate scheduler, and a weight decay of 0.0001. All models are trained for 120 epochs following ResNet (He et al., 2016), unless otherwise stated.
4.1 ABLATION STUDY
We empirically validate FACS design practices by performing the most pertinent ablations possible. ResNet-50 is adopted as the baseline for this purpose. To begin with, we first analyze the effect of changing the activation function in the rrosscchannel information blending stage, and then examine the effect of placing a BatchNorm prior to the sigmoidal activation. Further, we verify the behavior of proposed channel fusion strategies, and also the effect of varying fusion threshold Tof-
Fusion Activation. The channel fusion stage utilizes the fusion probability p produced by the sigmoidal layer. Given that the value of lies in the interval [0. 1]. we wished to examine the behavior of FACS if this range is achieved via different activation function. For this purpose, we select TanH function which natively squeezes the input into range [-1, Therefore, we rewrite the mathematical expression to 0.5 * (1+TanH) in order to place the output of TanH into the desired range of [0, 1). We replace the sigmoidal activation with the above expression and retrain the network. The corresponding results are shown in the top section of Table 1. The measures reveal that sigmoidal activation is superior TanH activation for the case of FACS.
BatchNorm in Global Context A Aggregation. Out of curiosity, we also analyze the behavior of FACS module by placing a BatchNorm (loffe & Szegedy, 2015) after the 1 x 1 convolution, because the squeeze layer in the baseline method is also followed by a BatchNorm layer. We observe that BatchNorm negatively impacts the performance (second section, Table 1).
Effeet of Fusion Threshold (Tef). The hyperparameter Tef is evaluated against three values (0.0, 0.5, 1.0). In accordance with the Eq. 2, Tef = O corresponds to Avg operator, Tof = 1.0 corresponds to Max operator regardless of the value p. Whereas Tef = 0.5 offers equal opportunity to the Max and Avg reduction operators which is adaptively taken care of by the value of p. We present an ablation over the aforementioned three values of Tef- The third section of Table holds the corresponding results. It is observable that Tef 0.3 attains the best performance, which is the case when the network has the flexibility to choose between the two reduction operatórs adaptively. Hence, in the subsequent experiments, we use Tef 0.5 for threshold based fusion. One may argue that there is marginal accuracy difference for different Tay values, so why not use a prefixed operator? As stated earlier in See 3.3, in that case p becomes of use, and corresponding conv layer will have to be removed, which translates to (Hussain & Hesheng, 2019), resulting in loss of parameters and accuracy (Table-2).
5

---

Under review as a conference paper at ICLR 2023
Table Ablation study of FACC-ResNet-S. Top-1 Accuracy on ImageNet.
Ablation
Parameter
Top-1 Accuracy
Sigmoid Tanll
76.77%
Fusion Activation
76.39%
Butch-Norm
X
76.77%
/
76.44%
0.0
76.58%
Tar
0.5
76.77%
1.0
76.54%
Threshold based Fusion (E5q. 2)
76.77%
Channel Fusion Stratogy
Probabilistic Weighted Fusion (Eq. 3)
76.52%
Channel Fusion Strategy. We train the network for both of the fusion strategies. Our findings suggest that threshold based fusion is superior to the weighted fusion, as shown in the bottom section of Table Although, the effect is marginal, faster as compared to Eq. (3). It is because, Eq. (2) requires atmost one operator to be computed based on the value of Tef, whereas Eq. (3) requires both of the operators to be computed.
the context of implementation, Eq. (2) is easier and
4.2 IMAGENET CLASSIFICATION
To the best of our knowledge, computationally efficient squeeze operation has not been discussed in the literature, hence due the lack of existing algorithms this problem, the most suitable evaluation possible is with the baselines only. Nonetheless, we could find (Hussain & Hesheng, 2019) which is only the aligned work. We have compared our approach with this approach.
4.2.1 FACS-RESNET
Table 2 shows evaluation of FACS when plugged into ResNet. FACS indeed achieves computationally efficient squeezing, as indicated by the ~ 23% reduction in FLOPs of all the FACS variants. Though improving the accuracy is not our goal, FACS also boosts the accuracy significantly i.c. 0.47%. 0.45%, 0.44% for FACS-50, 101. and 152 respectively. To our surprise, FACS-ResNet-100 surpasses the baseline R55 by 18% in accuracy and 47% in FLOPPs, indicating a huge improvement.
FACS-ResNet vs Depthwise/Channe Pooling.
terms of comparison, closely matching ap- proach is depthwise pooling (DWP) (Hussain & Hesheng, 2019) which is a parameterless method. We endow ResNet-50 with DWP and retrain the network with identical settings of MACS-ResNet. 50. Table 2 shows the corresponding comparison. It can be noticed that DWP drastically reduces the number of parameters of the baseline ResNet which results in a huge accuracy drop 1.68%. Although, there is significant parameter gap between FACS/ResNet and DWP, therefore, at first glance the comparison might seem unfair. However, our intention to conduct this experiment is to demonstrate that FACS does not incur any reduction in parameter while maintains the accuracy.
Reduction Ratio (R). It is important to note that the reduction ratio R is not hyperparameter of FACS. It can be potential source of confusion when discussing (Hu et al., 2018; Woo et al., 2018) which employ two reduction ratios: one for the squeeze layer (R) and one for the SE CBAM modules (r). The value of R is independent of the FACS, SE or CBAM since it is the hyperparameter of the baseline ResNet where it set to 4. Likewise, R is also set to 4 the FACS variants. On the other hand, the reduction ratio r is set to 16 in the SE unit of (Hu et al., 2018) and the shared MLP of the (Woo et al., 2018), 1)), but is irrelevant to FACS.
Although ablation on R is out of scope of this paper. yet we examine the impact of varying R on the performance of ResNet and FACS variants. To evaluate, we perform an additional experiment by setting R = 8 whose outcomes are displayed in Table 2. It can be noticed that with R = 8, MACS-ResNer performs better by 0.81% with almost 25% lesser FLOPPs which huge in itself. In addition, the accuracy gap between ResNet for R = 4 and R 8 is 2.6 2.64%, while this gap reduces to 2.30% for FACS at significant 56% reduction in the computation cost. It shows the robustness of FACS towards parameter reduction and improved generalization over large scale ImageNet dataset by learning richer data representations. further signifies the vast scope of FACS in edge devices, especially during network customization for required purpose.
FACS in Conjunction With SE. When used in conjunction with SE (Hu et al., 2018) in ResNet-50. FACS reduces the computational demands of baseline DE-ResNet-50l while performing better, Table 2.

---

Under review as a conference paper at ICLR 2023
Table 2: FACS ResNet. FACS better for a negative 'AAce%' and positive "A#FLOPS%'
Architecture
R
AParams
#FLOPs
Top-1
Top-5
AAcc%
AFFLOP%%
DWP-ResNet-50
4
19.6M
2.828
74.62%
92. 12.10%
ResNet-50 at. 2016)
25.5M
76.30%
92.69%
PACS-ResNet-500
4
25.5M
3.18B
76.77%
93.34%
-0.47%
22.8%
ResNet- (Ile al., 2016)
44.5M
7.85B
77.21 %
93.61%
FACS-ResNet-10
44.5M
6.05B
77.96%
93.91%
-0.45%
22.9%
ResNet- 152 (Ile et al., 2016)
4
60.1M
11.58B
77.78%
93.80%
FACS-ResNet-152
60.1M
8.91B
78.12%
94.02%
-0.44%
23.0%
ResNet-50
12.3M
1.85B
73.66%
91.59%
FAC5-ResNet-50
12.3M
1.398
74.47%
92.07%
-0.81%
24.8%
SE-ResNet-50 (Hu et 2018) SE:PACS-ResNat-00
28,
4.138
76.71%
93.38%
28.0M
3.19B
76.95%
93.40%
-0.24%
22.8%
Table 3: Latency analysis of FACS vs ResNet @224 x 224, FP32 precision, averaged over 25 runs.
Compute Platform
Cores
Compute power
Res-50
FACS-50
Res-101
FACS-101
Res-152
FACS-152
NVIDIA A410
10752
37.00 TFLOPs
7ms
Gms
11ms
10ms
15ms
14ms
NVIDIA KUX208UTI
4352
13.45 TFLOPs
8ms
6ms
14ms
12ms
17ms
15ms
NVIDIA GTX1080TI
3584
11.45 TFLOPs
9ms
7ms
13mx
12ms
17ms
15ms
NVIIIA Jetson NX
384
1,000 TFI OP
48ms
40ms
75ms
64ms
100ms
85ms
NVIDIA Jetson Nano
128
0.235 TFLOPs
1400m
130ms
230ms
200ms
320ms
280ms
Throughput & Latency. The ~ 23% FLOPs reduction directly reflects increased throughput as training time reduces from 4 to 2.5 days for ResNet-50, to 3 days for ResNet-1001, and 6 to 4 days for ResNet-152 on an 8x NVIDIA 1080Ti system. Additionally, huge latency improvements are also seen on edge computing devices. Table 3 shows the latency analysis on a number of GPUs with varying compute capabilities. It is done in order to demonstrate the portability of FACS across devices with varying computational power: The first three are high power GPUs among which the first is compute specialized, second and third are gaming GPUs with former powerful than the latter. On the other hand, the last two are edge computing devices which are far less powerful.
The difference in latency is mostly attributed to the variation in the number of computing elements or cores. Theoretically, more cores should run a network faster, however owing to the sequential linking of layers, layer must wait until preceding ones finish. This causes similar latency for the first three GPUs. However, for such devices, considerable gains are seen during batched inference and training, which is measure of throughput and, is reflected via days-long reduction in training time.
In contrast, the impact of FACS is more pronounced on edge devices, where the cores are a scarce resource. On Jetson-NX, FACS-50 is 16% faster then ResNet-50, FACS-101 is 14% faster, and FACS-152 is 15% faster. On Jenoon-Nano, FACS-50 is 7% faster then ResNet-50, FACS-101 is 13% faster, and FACS-152 is 12% faster. Notably, these speed-ups are huge which can be further enhanced via half-precision (FP16) inference, roughly doubling the speed. Considering the extensive usage of edge devices in real-time applications, the aforementioned enhancements are quite advantageous.
Dynamic Pruning (Gao et al., 2018) FACS Although dynamic pruning is not applicable to channel squeezing, we customize a recent approach "FBS" (Gao et al., 2018) for this purpose. FBS works by subsampling channels from the input via global pooling followed by convolution to compute salieney, and then keeping top-k channels based on that, while turning off the remaining ones by setting all the values to zero. Notice that, in FBS, both input and output tensors lie in RCxHxW To adapt it for our use, we keep the top-k channels, where equals to C/R. In this case, FBS outputs a tensor E R xxx instead of E CCx which is its default behavior.
During training, we see convergence issues. This happens because when keeping only top-k channels, the intermediate channels are dropped based on the input which can be different for different input. When this tensor is passed to the subsequent convolution, due to dynamically changing position of the channels in the output, the convolution struggles to learn the relation between the channels, because channel indexing is lost during choosing top-k. This problem does not occur in naive FBS. Secondly, FBS requires an additional convolution to compute channel saliency which has more parameters in contrast of FACS. It leads to roughly 48% rise in the parameters i.c. from 24.4M to 37M, which is not desirable. This clearly indicates that pruning methods can not complement FACS, which further signifies novelty and utility of FACS.
7

---

Under review as a conference paper at ICLR 2023
Table 4: FACS-VGG vs VGG on ImageNet.
Architecture
&Params
HFLOPss
Top-1
Top-5
AAce
AMFLOPs
VGG-16-1 161661x, x
10.3M
10.08
74.29% 74.82%
91.86%
FACS VGG-16
10.3M
9.3B
92.16%
-0.53%
7.0%
VGG-16-1 x FACS-VGG-BE
7.4M
6.7B
72.59%
90.88%
7.4M
6.3B
72.71%
91.01%
-0.12%
6.0%
Table 5: FACS for MobileNet-v2 MobileNet-v2 evaluation ImageNet.
Architecture
BatchNorm (lolfe & Szegedy, 2015) #Params
4FLOPs
Top-1
Top-5
AAce
AFFLOPS
MobileNet-r2 (Sandler et 2018)
3,50M
0.328
71.80%
90. 19%
MCCS-M
3.5UM
0.25B
89. 89.85%
0.67%
21.8%
FACS MobileNet.v2 FACS-MohileNet-22
Pre
3.50M
0.25B
71.00%
89.75%
0.80%
21.8%
Post
3.50M
0.25B
70.79%
89.50%
1.01%
21.8%
MobileNet. (Sandlerr et al., 2018)
25.2M
4.65B
74.48%
92.09%
X Pre
25.2M
4.58B
74.57%
92.07%
-0.09%
1.2%
FACS Mo
25. 2M
4.58B
74.65%
92. 11%
-0.17%%
1.2%
PACS-MobileNee-v2
Post
25.2M
4.58B
74.52%
92.17%
00.04%
1.2%
4.2.2 FACS-VGG
We select the popular VGG-16 variant of VGG series and upgrade it to VGG-16-1 x model with BatchNorm (loffe Szegedy, 2015) in all the layers. We compare this model with its FACS variant. The experiments are conducted for two values of reduction ratio R i.e. {2, 4). the results for which are shown Table 4.
can be observed that the FACS is consistent as computation reduction ~ 6% while accuracy is improved significantly for different values of R. Noticeably, relative gain w.r.t. ResNet is smaller (23% vs 6%), it is so because in this architecture most computations are packed into 3 3 convolutions (90%) instead of 1 (6%). Nonetheless, potential of FACS is huge in VGG like architecture. especially for low power applications and edge devices because narrower models of VGG are quite popular for real-time applications (Kumar al., 2020).
4.2.3 FACS-MOBILENETT-22
Mobile networks are not our point of consideration because of their limited representation power and complexity, as mentioned in Sec. 2. However, we analyze how FACS behaves with mobilenets which employ inverted-residual bottlenecks consisting of depthwise separable convolution networks. We choose MobileNet-v2 (Sandler et al., 2018) for the evaluation purpose and train for 120 epochs.
The experimental results are reported in Table 5. We observe that for depthwise separable convolution, FACS performs slightly inferior. We hypothesis that the accuracy drop is due to the lack of inter- channel communication in depth-wise separable convolutions. In MobileNet-v2, this communication performed by the squeeze layer in addition to performing the task of reducing the number of channels. However in FACS, this is performed by global pooling and the convolution layer which operates on descriptor (less information) instead the tensor (more information) which according to our hypothesis becomes the reason of slightly reduced accuracy.
To validate our hypothesis, replace all the depthwise separable convolution by fully connected convolutions MobileNet-v2. In this case, we observe improvements in the accuracy, as shown in the second section of Table 5. This verifies our previously made hypothesis that devoid of inter-channel communication limits the performance of FACS. However, in this case the FLOPs improvement drops which is expected. It is because, inverted residual design increases the number of channels which when fed to a fully connected 3 x 3 convolution, leads to most of the network computations packed into these layers (95%). Hence, effect of FACS diminishes. This experiment for hypothesis verification only because inverted bottlenecks with fully connected 3 x 3 are not used in practice due to exponential rise in computations.
In addition to the above, we also performed ablations on MobileNet-v2 by placing BatchNorm (Ioffe & Szegedy, 2015) at the input and output of FACS. We observe that BatchNorm slightly lowers the performance depthwise convolutions while for the fully connected case, pre- BatchNorm marginally improves the performance. Conclusively, we say that FACS performs par for depthwise separable convolutions, but 21% reduction in FLOPs can be traded for an accuracy drop of 0.67% which is negligible. While FACS performs better for fully connected convolutions inverted-residua units.
8

---

Under review as a conference paper at ICLR 2023
Table FACS vs ResNet on classification, and semantic segmentation transfer learning.
Architecture
CIFAR-10
CIFAR 100
CltyScapes
ResNett-0 (He et 2016) FACS ResNet-50
95.57% 95.67%
81.60% 82.22%
PSPNet (Zhao et al., 2017) + ResNet-101 (He et al., 2016)
78.4%
PSPNet (Zho al., 2017) FACS ResNet-10) 101
79.
Image
Image
RecNet
VGG
FACS
FACS
Figure 5: GradCAM visualizations. More red shows more confidence for pixel belong class.
4.3 TRANSFER LEARNING
To analyze the generalization of FACS across datasets, we perform transfer learning from ImageNet CIFAR-10 and CIFAR-100. Each of the dataset consists of 50K training and 10K test images. For training, we finetune FACS-ResNet-50 and ResNet-50 pretrained over ImageNet. The training protocol for both the datasets remain identical to that of ImageNet except 200 epochs. Table 6 shows the classification results. It can be seen that, surprisingly. FACS- RecC-R00 outperforms the baseline by 0.10% on CIFAR-10 10 and by 0.62% on CIFAR-100 at lower FL.OP requirements.
In addition, we evaluate FACS for challenging downstream task of semantic segmentation. We use the popular approach (Zhao et al., 2017) and replace the backbone with FAC'SResNNe-101.. Consistently, FACS variant outperforms the baseline both in terms of FLOPs efficiency and accuracy by 0.7% units mloU (mean intersection over union), indicating that FACS indeed transfers better across tasks and datasets.
4.4 GRADCAM VISUALIZATION
As FACS employs global context in one of its stages and also choose the reduction operators adaptively, it encouraged us to analyze the behavior of layer activations in the network corresponding to groundthruth class.
this, we use GradCAM (Selvaraju et al., 2017) that utilizes gradients to visualize the importance of spatial activations. GradCAM is given a label for which the attended regions in the feature maps are computed. This feature of GradCAM utilized to visualize and understand that how FACS attends the spatial regions in comparison to the baseline networks. It gives us the qualitative explanation that despite the reduction in the computations, why FACS is performing
Figure shows the GradCAM analysis for ResNet and VGG. It can be seen that FACS significantly improves the attended regions of the target class relative to the baselines. Also, in case of images with multiple instances of an object, FACS focuses each instance strongly. It indicates that FACS enhances a network's ability to generalize better by learning to emphasize class-specific parts.
CONCLUSION
In this work, we introduce novel module called "Fast Adaptive Channel Squeezing" (FACS) which redefines the manner in which channel squeezing is accomplished in CNNs. It functions by exploiting global spatial context and blending cross-channel information in order to carry out computationally efficient channel squeezing. We plugged the FACS module into three distinct CNN architectures: ResNet, VGG, and MobileNet-v2, and presented comprehensive ablation study including ImageNet classification results and transfer learning semantic segmentation task. Our experimental study suggests that MACCS-nnanced CNNs are roughly 23% faster (lower FLOPs) while being more accurate in the same number of parameters while generalizing well across datasets and tasks. We show that the reduction in FLOPs cuts a network's 's training time significantly, and also demonstrate that FACS results in lower latency of network which speeds-up inference edge devices.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229 Springer, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13733-13742 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint vrXiv:2010.11922 2020.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05331, 2018.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04880 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and- anneeeettand-xxiittttinnn networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018
Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. CVPR, 2017.
Abid Hussain and Wang Hesheng. Depth-wise pooling: A parameter-less solution for channel reduction of feature-map in convolutional neural network. 2019 IEEE International Conference on Real-time Computing and Robotics (RCAR), pp. 299-304. IEEE, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. pp. 448-456. PMLR, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105. 2012.
Ashish Kumar and Laxmidhar Behera. Semi supervised deep quick instance detection and segmenta- tion. In 2019 International Conference on Robotics and Automation (ICRA), pp. 3325-8331. IEEE, 2019.
Ashish Kumar, Mohit Vohra, Ravi Prakash, and Laxmidhar Behera. Towards deep learning as- sisted autonomous uavs for manipulation tasks in gps-denied environments. In 2020 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1613-1620. IEEE, 2020.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608. 03998, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perii peghomerformence deep learning library. Advances in neural information processing systems, 32, 2019.
10

---

Under review as a conference paper at ICLR 2023
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-tine rall-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Nick Schneider, Florian Piewak, Christoph Stiller, and Uwe Franke. Regnet: Multimodal sensor registration using deep neural networks. In 2017 IEEE intelligent vehicles symposium (IV), pp. 1803-1810. IEEE, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Sifre and S Mallat, Rigid-motion scattering for image classification. arxiv 2014, arXiv preprint arXiv:1403.1687.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.11550 2014.
Mingxing Tan and Quoe Le. Efficientner: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14. PMLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Sanghyun Woo, Jongchan Park, Youn-oung Lee, and So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference computer vision (ECCV), pp. 2018.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890, 2017.

---

Under review as a conference paper at ICLR 2023
APPENDIX
A COMPUTATIONAL COMPLEXITY
The primary intent of the FACS module has been to reduce the computational complexity of squeeze operation. Therefore, it is necessary to discuss the computational requirement associated with FACS. However for better understanding, first discuss the FLOPs of different kinds of layers.
A.1 CONVOLUTION
Consider a convolution layer having N kernels and an input feature map X E RCXH The size of each kernel can be given by C k x k. FLOPs for convolution operation are determined in terms of Musion-Vulti-Addiiin (FMA) instructions, therefore the computational demands of a convolution layer can be given as:
#FLOPs = H x X
A.2 BATCHNORM
The BatchNorm Ioffe & Szegedy (2015) operation is performed per spatial location and can be given as & (X - + 3. can implemented in three FLOPs i.e. first for computing X - μ., second for 1/0 and last as FMA with B. general, a is stored as o², therefore, it requires to compute square-root of の² to obtain Overall, it takes four FLOPs implement BatchNorm operation per spatial location. Thus, total number FLOPs for a BatchNorm layer can be given as:
#FLOPs 4 x C x H W
(5)
Optionally, during inference, BN can be fused with Conv operation where convolution is followed by BN, but we remain agnostic to such cases to account for the training phase and other architectures.
A.3 RELU
A ReLU operation is given by Y X for X > 0 and Y 0 for X < 0. It simply requires comparison instruction, leading to total number of FLOPs given by:
#FLOPs = Hx
(6)
A.4 SIGMOID
Sigmoid operation is given by = 1/1+exp x, It can be implemented in four FLOPs. Therefore, the total FLOPs for a Sigmoid layer can be given by:
#FLOPs 4xCxH
A.5 GLOBAL POOLING
Apart from the above layers, in an FACS module, a global pooling operation also performed. There several ways to implement global pooling operation, however, the most common is by using matrix multiplication routines and FMA instructions. The whole channel of a feature map can be considered a vector of size H W which can be reduced scalar by taking its dot product with a vector whose all of the elements equals to one. Hence, the total number of FLOPs for the global pooling operation can be given by:
#FLOPs CxH
(8)
A.6 CHANNEL FUSION
Channel fusion operates on chunk of & channels. For Max operation, (k 1) compare instructions, while for an Avg operation, (k - 1) FMA instructions are required. Thus, the total number FLOPSs for channel fusion can be given by:
#FLOPs (k 1) Hxx
(9)
12

---

Under review as a conference paper at ICLR 2023
Based the several equations developed above, the computational complexity of residual bottleneck and the FACS block can be calculated.
A.7 EFFECT OF FACS ON COMPUTATIONAL COMPLEXITY
In the baseline method, the squeeze layer operates upon X E RC which requires C/R x C' x И x W FLOPs. Whereas in FACS, the global context aggregation requires C' H x W FLOPs, cross-channel information blending requires C/R C' FLOPs. and channel fusion requires C/R x (R 1) x H x W FLOPs.
As example, consider input tensor X R12x6x6 to squeeze layer kernels of size 1 1. With R = 4, the N 3. From the equations below, total number of FLOPs for a squeeze layer equals to 8475.
# Conv FLOPs = 5 x 5 x 3 x 12 x 1 1 = 900
(10)
BN FLOPs 4
5 =300
(11)
RReL._FFLOPS = 3 x 5 x 75
(12)
On the other hand, based on the following equations, the FLOPs for the FACS module with == equals to 811.
Pooling FLOPs 12 x 5 x 5 = 250
(13)
#Conv FL.OPs
12 1 36
(14)
# Sigmoid FLOPs = 4 x 3 X x 1 12
(15)
#Fusion FLOPs
3 5 225
(16)
In the above example, the baseline squeezing method requires 1275 FLOPs, whereas FACS requires only 523 and 748 FLOPs for threshold based and weight fusion respectively. In the similar manner, achieve huge gains when FACS is plugged into the existing networks which are discussed in the experiments. For the calculation of FLOPs, kindly see above.
A.8 EFFECT OF FACS ON MEMORY
Despite the computational benefits, FACS does not introduce any memory overhead. The total memory required by the baseline squeeze operation can be given by: #M C/4 x H x W. On the other hand, the memory required for FACS is given by: #M C + C/ 4 C/4 x H x W. We can see that there is negligible increment in the memory footprint i.c. from 0.75 x C x H x W to 0.75 H W + 1.25C. For FP32 precision, the raw memory footprint will be x M.
B CODES AND IMPLEMENTATION
We will opensource the codes and all the pretrained models for the framework in PyTorch (Paszke al., 2019). In order to better understand the concept, we have included our GPU implementa- tion of channel fusion layer in the supplementary material. The corresponding file is named as facs_ lacs__aaerr.cm cu which consists GPU implementation of the fusion formulations described in the paper. Other files facs_ Laerr... . cpp, facs sacc_layer.._ py also provided which holds pytorch class implementation FACS. We have not provided the network training and inference code which will be released later.
also contains code for the new experiment related to FBS in three files: Sacs_fbs_lass cu which consists of GPU implementationa along with helper files facs fbs layer cpp, facs fbs loos_layer.py
C GPU CONSIDERATIONS
The implementation of FACS operator is quite straightforward and fully parallelizable. For reference. we also have provided the GPU implementation in the supplementary material. The computations of fusion probability and the fused feature map are entirely parallelizable because they are pointwise operations. The whole code can be implemented in merely 10 - 15 lines of a CUDA kerne and other parallelization device.
13


Reviews:
[{"review_id": "tS_iKRAl0n", "reviewer": "Reviewer_Ywmp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a FACS network to reduce the computation cost of deep learning models by redefining the module in channel squeezing. The original channel squeezing module (1x1 convolution) is amended by adding the global pooling and designing the channel fusion strategies. ", "summary_of_the_review": "As mentioned in 'Strength And Weaknesses', the proposed method novelty is not sufficient. Additionally, it also lacks of theoretical analysis. So I tend to give the 'marginally below the acceptance threshold'.", "strength_and_weaknesses": "Pros:\n\nThe proposed FACS method is able to reduce the computation cost and improve performance.\n\nCons:\n\n(1) In my opinion, the global pooled feature will have the same effect on the input feature (with respect to performance), compared with the baseline squeezing. So the performance improvement should be caused by the channel fusion strategies. The authors are suggested to add one ablation study to show the effect of channel fusion strategies (e.g., a single type of pooling).\n\n(2) The operations in the FACS block are very similar to that in SEnet (1 x 1 Conv on c x 1 x 1 feature is the same with the Fc layer).\n\n(3) For Sec 3.3.1 and Sec 3.3.2, why output channel from the channel squeezing can be aligned with the channel-wise avg/max pooling?\n\n(4) Lack of theoretical analysis of why the FACS and channel fusion module are useful.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eom5Zy1jV_C", "reviewer": "Reviewer_9T37", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In order to speed up the channel squeezing task, this paper proposes the Fast Adaptive Channel Squeezing (FACS) module to replace the conventional 1x1 convolution operator used in channel reduction. The key benefit of FACS is that it neither alters the number of parameters nor affects the accuracy of a given network. Experiments show that FACS reduces FLOPs yet keeps model performance.  ", "summary_of_the_review": "The primary concern of this paper is its technical novelty. Several experiments support the proposed FACS, yet no theoretical explanation or invention of new operations. Therefore, it is doubtful if the proposed method could reach the high standard bar of ICLR.", "strength_and_weaknesses": "[Strengths] \n+ This paper is well-written and easy to follow. Most operations are illustrated with corresponding figures and formulas.\n+ The manuscript conducts several experiments.\n\n[Weaknesses] \n- As the authors sensed, the reader is intuitively aware that the proposed FACS indeed similar to the existing SE-Net [8] and CBAM [25]. The FACS employs the squeeze and channel reduction steps from SE-Net and then excites each channel with the spatial attention as CBAM. Therefore, a supported theory to further explain why composing these operations in the FACS is sufficient and what operations cause its benefits are needed to clarify the work's contribution and novelty.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZEzljidVbrQ", "reviewer": "Reviewer_ZY5e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to reduce the computation cost of deep learning model by reformulating the channel squeezing operation. Specifically, the 1x1 conv operation is replaced by firstly channel-wise average pooling the input feature, then calculated the called 'fusion possibilty'. Finally, the input channels are squeezed to be target size with the output 'fusion possibilty'.", "summary_of_the_review": "The proposed method is far different from the exiting deep learning working scheme, and lacks of corresponding explanation and analysis, which is hard to believe the reported performance.", "strength_and_weaknesses": "Weaknesses:\n\n1. The reference style is problematic.\n\n2. In Sec2.1, the comparison on number of training epoch dose not equal to the comparison of training efficiency, since the compared models are different.\n\n3. Most of the 1x1 conv layers in ResNet are used for increasing the non-linearity while preserving the resolution of feature, it cannot be treated as channel squeezing.\n\n4. The 'Global Context Aggregation' operation collapse the representation into single dimension, which must leads to spatial information loss.\n\n5. In Sec 3.3.1, why the lower possibilities correspond to the average pooling while the higer ones correspond to max pooling? Any analysis and experiments for this operation?\n\n6. In Sec3.3, the proposed combination manner is totally different from the existing one, since the combination of max and mean pooling should cover only a small region of the previous representation combination manner. However, there is no analysis and explanation to reason the proposed manner, which is hard to follow and understand.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "2r6nn6o9UC", "reviewer": "Reviewer_LDvY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a run-time channel squeezing technique. The proposed technique mainly works on the convolutional block and can serve as a plug and play component to many CNNs. The proposed methods does not keeps all possible parameters but can reduce the number of FLOPs to enable fast inference on low-power devices.", "summary_of_the_review": "This paper proposed an intersting idea, the fact that the proposed method can serve as a plug-and-play component is interesting. However, I do not think the authors have presented a detailed comparison to many existing works in this domain, and this generally affects the quality of this paper and makes it hard to gauge the real contribution. ", "strength_and_weaknesses": "## Strength\n\n- The proposed run-time squeezing scheme, although similar to dynamic pruning techniques, is used as a plug-and-play component. This seamless integration can directly improve the runtime performance of many existing CNNs.\n- The authors reported latency numbers on a range of hardware platforms.\n- The design choices are well justified in Section 4.1\n\n## Weakness\n\nThe major weakness of this paper is on its evaluation, it is not clear to me how this proposed method compares to other published methods.\n\n- This paper does not provide a direct comparison to many existing techniques in the field of dynamic pruning [1, 2, 3]. These dynamic channel pruning techniques are closely related to this work.\n- In all the results presented, the authors are mainly comparing to baselines. For example, Table 2 mainly compares a modified ResNet50 with FACS-ResNet50. This does not really provide me with a full picture, what if the ResNet50 is modified further to have the same number of FLOPs? What is the top-1 accuracy gap in this case?\n- The proposed technique performs a lot better on ‘old-school’ CNNs like VGG and ResNet. On more trimmed model such as MobileNet, the proposed method seems to have diminishing returns. Also models like MobileNet-V3 and EfficientNet might need to be considered.\n\n[1] Dynamic Channel Pruning: Feature Boosting and Suppression\n\n[2] Channel gating neural networks\n\n[3] Boosting the performance of cnn accelerators with dynamic fine-grained channel gating", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject