Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a run-time channel squeezing technique. The proposed technique mainly works on the convolutional block and can serve as a plug and play component to many CNNs. The proposed methods does not keeps all possible parameters but can reduce the number of FLOPs to enable fast inference on low-power devices.
strength_and_weaknesses: ## Strength

- The proposed run-time squeezing scheme, although similar to dynamic pruning techniques, is used as a plug-and-play component. This seamless integration can directly improve the runtime performance of many existing CNNs.
- The authors reported latency numbers on a range of hardware platforms.
- The design choices are well justified in Section 4.1

## Weakness

The major weakness of this paper is on its evaluation, it is not clear to me how this proposed method compares to other published methods.

- This paper does not provide a direct comparison to many existing techniques in the field of dynamic pruning [1, 2, 3]. These dynamic channel pruning techniques are closely related to this work.
- In all the results presented, the authors are mainly comparing to baselines. For example, Table 2 mainly compares a modified ResNet50 with FACS-ResNet50. This does not really provide me with a full picture, what if the ResNet50 is modified further to have the same number of FLOPs? What is the top-1 accuracy gap in this case?
- The proposed technique performs a lot better on ‘old-school’ CNNs like VGG and ResNet. On more trimmed model such as MobileNet, the proposed method seems to have diminishing returns. Also models like MobileNet-V3 and EfficientNet might need to be considered.

[1] Dynamic Channel Pruning: Feature Boosting and Suppression

[2] Channel gating neural networks

[3] Boosting the performance of cnn accelerators with dynamic fine-grained channel gating
clarity,_quality,_novelty_and_reproducibility: ## Quality

This paper provides an interesting idea about squeezing the channels at run-time in a dynamic fashion. This dynamic operation helps the networks to preserve its full capacity but can use a smaller number of FLOPs for the given task. The evaluation of this proposed method is concerning, it only makes a comparison to baseline models, this paper has ignored the entire field of dynamic pruning. These flaws in evaluation have a great impact of the quality of this paper.

## Clarity

- This paper has used some non-standard reference styles, and also its cross reference style looks a bit abnormal to me.
- Figure 4 is very confusing. I would prefer you label the input and output dimension of each block, and give the block a name and say the weights (eg. the same style in Figure 1) are having these shapes. The illustration is connecting weights together, which is fairly strange.

## Novelty and Reproducibility

It is hard to judge the novelty of the proposed paper without the authors giving me a full explanation of how it is different from other dynamic pruning techniques.
summary_of_the_review: This paper proposed an intersting idea, the fact that the proposed method can serve as a plug-and-play component is interesting. However, I do not think the authors have presented a detailed comparison to many existing works in this domain, and this generally affects the quality of this paper and makes it hard to gauge the real contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In order to speed up the channel squeezing task, this paper proposes the Fast Adaptive Channel Squeezing (FACS) module to replace the conventional 1x1 convolution operator used in channel reduction. The key benefit of FACS is that it neither alters the number of parameters nor affects the accuracy of a given network. Experiments show that FACS reduces FLOPs yet keeps model performance.  
strength_and_weaknesses: [Strengths] 
+ This paper is well-written and easy to follow. Most operations are illustrated with corresponding figures and formulas.
+ The manuscript conducts several experiments.

[Weaknesses] 
- As the authors sensed, the reader is intuitively aware that the proposed FACS indeed similar to the existing SE-Net [8] and CBAM [25]. The FACS employs the squeeze and channel reduction steps from SE-Net and then excites each channel with the spatial attention as CBAM. Therefore, a supported theory to further explain why composing these operations in the FACS is sufficient and what operations cause its benefits are needed to clarify the work's contribution and novelty.

clarity,_quality,_novelty_and_reproducibility: This paper is well-written and hence has good clarity and reproducibility.

The technical novelty is limited, as shown in [Weaknesses].
summary_of_the_review: The primary concern of this paper is its technical novelty. Several experiments support the proposed FACS, yet no theoretical explanation or invention of new operations. Therefore, it is doubtful if the proposed method could reach the high standard bar of ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a FACS network to reduce the computation cost of deep learning models by redefining the module in channel squeezing. The original channel squeezing module (1x1 convolution) is amended by adding the global pooling and designing the channel fusion strategies. 
strength_and_weaknesses: Pros:

The proposed FACS method is able to reduce the computation cost and improve performance.

Cons:

(1) In my opinion, the global pooled feature will have the same effect on the input feature (with respect to performance), compared with the baseline squeezing. So the performance improvement should be caused by the channel fusion strategies. The authors are suggested to add one ablation study to show the effect of channel fusion strategies (e.g., a single type of pooling).

(2) The operations in the FACS block are very similar to that in SEnet (1 x 1 Conv on c x 1 x 1 feature is the same with the Fc layer).

(3) For Sec 3.3.1 and Sec 3.3.2, why output channel from the channel squeezing can be aligned with the channel-wise avg/max pooling?

(4) Lack of theoretical analysis of why the FACS and channel fusion module are useful.



clarity,_quality,_novelty_and_reproducibility: The reference style is wrong. The write-up is not very clear. 

The novelty is limited 

The authors provided the code in supplemental materials
summary_of_the_review: As mentioned in 'Strength And Weaknesses', the proposed method novelty is not sufficient. Additionally, it also lacks of theoretical analysis. So I tend to give the 'marginally below the acceptance threshold'.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose to reduce the computation cost of deep learning model by reformulating the channel squeezing operation. Specifically, the 1x1 conv operation is replaced by firstly channel-wise average pooling the input feature, then calculated the called 'fusion possibilty'. Finally, the input channels are squeezed to be target size with the output 'fusion possibilty'.
strength_and_weaknesses: Weaknesses:

1. The reference style is problematic.

2. In Sec2.1, the comparison on number of training epoch dose not equal to the comparison of training efficiency, since the compared models are different.

3. Most of the 1x1 conv layers in ResNet are used for increasing the non-linearity while preserving the resolution of feature, it cannot be treated as channel squeezing.

4. The 'Global Context Aggregation' operation collapse the representation into single dimension, which must leads to spatial information loss.

5. In Sec 3.3.1, why the lower possibilities correspond to the average pooling while the higer ones correspond to max pooling? Any analysis and experiments for this operation?

6. In Sec3.3, the proposed combination manner is totally different from the existing one, since the combination of max and mean pooling should cover only a small region of the previous representation combination manner. However, there is no analysis and explanation to reason the proposed manner, which is hard to follow and understand.
clarity,_quality,_novelty_and_reproducibility: 1. The writing is not good.

2. Lacks of analysis and explanation.

3. Limited novelty.
summary_of_the_review: The proposed method is far different from the exiting deep learning working scheme, and lacks of corresponding explanation and analysis, which is hard to believe the reported performance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission introduces an approach called Fast Adaptive Channel Squeezing (FACS) which aims to carry out the channel squeezing operation in convolutional neural networks in a computationally efficient manner and without affecting capacity and accuracy. FACS operates in three stages on an input of shape [C, H, W]:

1. a global average pooling operation is applied to obtain a 1D descriptor of shape [C, 1, 1];
2. information is blended across channels through a linear mapping followed by an elementwise sigmoid nonlinearity, resulting in a 1D fusion probability vector of shape [C / R, 1, 1]; and
3. the input channels are partitioned into chunks of R channels and squeezed through a fusion operation (which operates independently for each spatial position) using the fusion probability vector, resulting in an output of shape [C / R, H, W].

Two fusion strategies are proposed:

1. threshold based function computes either the max or average over a given chunk of channels (depending on whether the corresponding fusion probability is below or above some threshold), and
2. weighted fusion computes a convex combination of the max and average using the corresponding fusion probability).

Empirical evaluation on three architecture families (VGG, ResNet, MobileNet-v2) and on three datasets (ImageNet, CIFAR-10, CIFAR-100) and FACS is shown to be around 23% faster (in terms of FLOPs) while offering a marginal accuracy improvement in most cases . Ablation studies are also performed to assess the effect of the fusion activation function, the interaction of FACS with batch normalization, the effect of the fusion threshold, and the impact of the channel fusion strategy. Finally, transfer learning and GradCAM visualizations are presented.

Reviewers note that the proposed approach is effective at reducing the computational cost while maintaining or improving performance (Ywmp), that it is easily applied to many CNN architecture families (LDvY), and that many experiments are presented to validate it (LDvY, 9T37). The main reviewer concerns revolve around baselines, comparisons against existing work, and technical novelty:

- Reviewer LDvY points out that the submission does not compare against dynamic pruning approaches, which in their opinion are closely related. The authors respond by clarifying that FACS' objective is to develop a channel squeezing method which can preserve model capacity while reducing computational cost, whereas dynamic channel pruning seeks to save on computation at inference time at the cost of accuracy reduction. They argue that pruning cannot be applied to channel squeezing, as it incorporates new parameters and modifies the network architecture, and that FACS and pruning are therefore orthogonal and cannot be compared. They mention an attempt to train a modified FBS pruning approach which discards channels not in the top-k instead of zeroing them out and report convergence issues and an increase in model parameters. Reviewer LDvY disagrees with the authors: from their perspective, both FACS and run-time pruning seek to preserve model capacity while reducing the computational cost, and pruning approaches do not necessarily need to be used to perform channel squeezing in order for them to be directly comparable. Both sides continue to argue their points further, but at this stage I should say that their thoughts have been made clear, and I agree with Reviewer LDvY. In the reviewer discussion, Reviewers 9T37 and Ywmp also agree. I see the importance of reducing computational requirements and maintaining model capacity, but I don't see why this necessarily has to be done through improved channel squeezing. FACS may be the first approach to more computationally efficient channel squeezing, but it's not the first work which looks into reducing the computational cost of a network while maintaining good accuracy. As such, it should be possible to compare model accuracies while controlling for FLOPs, and if as the authors claim pruning makes too large an accuracy sacrifice for the computational savings, FACS would be clearly shown to be superior in that regard. The analogy to SENet and its evaluation protocol put forward by the authors does not quite hold, either: SENet was used by the winning entry in the ILSVRC 2017 classification competition, and as a result was compared against a plethora of competing approaches (see Table 8 in the SENet paper). 

(continued in "Justification For Why Not Higher Score")
justification_for_why_not_higher_score: 
- Reviewer LDvY also points out that a relevant baseline to compare against would be a model of the same architecture family in which the capacity has been reduced so as to match FACS in terms of FLOPs. The authors counter that FACS is the only approach to perform efficient channel squeezing, and since they are interested in reducing the FLOPs, the only choice for comparison is the unmodified network architecture. They point to Table 2 for experiments on ResNet-50 where the reduction ratio has been set to 8 rather than 4 and which shows that FACS still comes out ahead. I agree with Reviewer LDvY, and from my perspective the authors' response does not address this point directly: Table 2 compares models while controlling for capacity in the form of number of parameters, whereas Reviewer LDvY asks about controlling for FLOPs. The ResNet-50 with a reduction ratio of 8 is not directly comparable to FACS-ResNet-50 in that regard because they require 1.85B and 3.18B FLOPs, respectively. The question of how much of an accuracy drop one would suffer by adjusting the numbers of channels in a ResNet-50 architecture so as to match the FLOPs in FACS-ResNet-50 is relevant, and it still stands.
- Reviewers 9T37 and Ywmp note the similarity in architectures between FACS and the existing SENet and CBAM architectures. The SE block performs global average pooling followed by a single-layer (dimensionality-preserving) nonlinear (ReLU) projection and multiplies the result channel-wise with the (unmodified) input. FACS performs global average pooling followed by a single-layer (dimensionality-reducing) nonlinear (sigmoid) projection and multiplies the result with the (channel-wise max-and/or-average-pooled) input. CBAM's channel attention module performs channel-wise max-pooling and average-pooling, passes both independently through the same MLP, then passes the summed outputs through a (sigmoid) nonlinearity and multiplies the result with the (unmodified) input. From Reviewer Ywmp's perspective, the most notable difference resides in the channel fusion strategy, and they would like to see a better characterization of this design choice. The authors reply that while all three share architectural elements, FACS' purpose is different, as it targets efficient channel squeezing, and it results in a network architecture that is different because it is parameter-preserving, unlike SENet and CBAM. They also provide additional experiments characterizing the design choices for the fusion mechanism. Reviewers 9T37 and Ywmp remain unconvinced regarding the submission's technical novelty. On this point, both the authors' and the reviewers' arguments have merit. The authors are right to point out that FACS tackles a new problem and that this constitutes a novel contribution, even if the mechanism through which it approaches the problem shares architectural elements with existing work. However, I understand the reviewers' unease about the way in which FACS is introduced and the authors' insistence on presenting it as a completely new approach. _(On that note, I should clarify that after reaching out to Reviewer LDvY regarding their comment on novelty quoted by the authors, Reviewer LDvY agrees with the other reviewers; what they were suggesting is that the authors simply acknowledge the similarities to previous work and focus on framing their work in that context rather than attempt to convince reviewers that their proposed approach is highly novel.)_ I think it would also be factually correct to present FACS as a channel squeezing approach which replaces the bottleneck with a SENet-like block and pairs it with a channel fusion mechanism to enable channel reduction (which is unaddressed in SENet). It would also be factually correct to say that FACS and CBAM share a design element of channel-wise average and max pooling, although FACS extends the concept to enable channel reduction (which again is unaddressed in CBAM). The authors should be more transparent on this in their discussion of related work.


Ultimately, I believe the reviewer and author disagreement on baselines, comparison against related work, and framing of the proposed approach stems from differing viewpoints rather than a lack of engagement with or understanding of the submission. Since the reviewers remain unconvinced by the authors' arguments, the submission does not quite meet the bar for acceptance.
Decision: Reject