Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies the channel-collapsed problem in CNNs using 'BN+ReLU' . The Channel Equilibrium block which consists of batch decorrelation branch and adaptive instance inverse branch are proposed to reduce the channel-level sparsity. Experiments on ImageNet and COCO demonstrate that the proposed CE block can achieve higher performance than the conventional CNNs by introducing little computational complexity. The author also discuss the relationship between the proposed method and Nash Equilibrium.

Pros:

+ The experimental results are impressive, the proposed block can improve the accuracy of CNNs while requires little additional computation cost.
+ This paper is well-written and easy to follow. The authors give a explicit explanation as well as prove of the proposed scheme. 

Cons:

- The motivation of this paper seems to be weak. The author argues that popular CNNs with 'BN+ReLU' have certain channels which would always output 0 for any input. Why not directly remove this channel to achieve speed-up? 
- Moreover, the author argues that 'BN+ReLU' block would lead to channel-level sparsity according to [1]. However, [1] says that this sparsity relies on weight decay. Figure 3 (d) also proves that the sparsity ratios of BN and CE are all 0 when weight decay is set as 0 (also notes that they achieve best accuracy when weight decay is 0). The results demonstrate that the higher accuracy of CE does not rely on its lower sparsity ratio.

In conclusion, the proposed CE is effective for achieving higher accuracy. However, the motivation and argument of the proposed method seems to be invalid, which prevents this work to be accepted.

[1] On implicit filter level sparsity in convolutional neural networks. CVPR, 2019.
review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new block that gives diminishing 

Pros)
(+) This paper is well-written and the idea looks interesting.
(+) The provided experiments look quite extensive. I like the experiments with both heavy and light network architectures to show the effectiveness of the proposed method.
(+) The authors provided plausible and sufficient backups for the claim.
(+) Connection with Nash equilibrium seems to be slightly overclaimed, but the attempt is novel and interesting.


Cons)
(-) The overall paper flow could be organized much better. I think some of the materials in the appendix needs to be placed in the main paper.
(-) Some of the notations in Section 3 need to be rewritten for better clarity. 
(-) The terminology such as channel-level sparsity, channel equalization, and equivalent lambda looks vague. I recommend the authors define them for clarity.

Comments) 
- I think the relationship between diminishing the sparsity on channels and improving the evaluation performance should be more addressed. If we change ReLU after Conv with other nonlinear functions such as ELU or PReLU, then the output hardly becomes zero, but as you know the performance usually gets worse. Then, how do you explain the necessity of reducing sparsities? 
- How did you measure the sparsity in a plain network in Figure 1 and Figure 3? Did you pick a single layer or gather all the layers' sparsity? Please specify the way of measuring it.  
- How did you plot the graphs in Figure 3.(a) and 3.(b). Did you pick a single channel randomly?
- To support the authors' claim, it would be better to show the sparsity ratio for a model (ResNet18, MobileNetV2, and so on) which shows better performance than each original one. 
- How did you determine the transformation of F in the AII branch? Please provide any intuitions why the authors choose the transformation like in eq.(9). 
- If CE-block actually conducts well right after a BN layer, then why CE block is attached only after the final BN layer in a bottleneck module? It would be better to provide any results by doing some studies when dealing with the intermediate BNs in a bottleneck module.
- I am wondering whether adaptive normalization techniques such as Instance-Batchnormalziation or Switchable Normalization methods could also give less sparsified features. Please clarify this by comparing with the proposed method.
- It seems that CE would have some extra computational costs compared with SE's. Please clarify why CE-module does have small computational costs.
- Please specify what MobileNetV2 has used. Looks like MobileNetV2x1.0 would be used.
- Why do you think the Top-5 accuracy of MobileNetV2 has not been improved?
- Can your method combine with ResNet with SE such as ResNet50SE? 
- Please clarify the way of measuring the sparsity ratio in Figure 3d.   Why did you consider the sparseness of a feature by measuring the gamma in Figure 4? There may exist a channel that contains large magnitude values then gamma cannot make the output close to zero.

About overall rating)
The theoretical backups for the authors' claim look sound, and all the experiments including the performance improvement with several models seem to support the effectiveness of the idea very well. Specifically, I like all the analyses and the connection with Nash equilibrium, which are very intuitive and may provide further insights to researchers in this field.  I think this paper is clearly above the standard of ICLR. If the authors could address all the concerns above and refine the paper with better readability, then I could increase the score. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors point out that CNNs can develop collapsed channels that limit their capacity. They propose to remedy this with batch decorrelation (BD), which focuses on ensuring that channels play an equal role in the feature map and are less likely to collapse. The claim is supported with experiments on CIFAR10, ImageNet and COCO.

Figures 1 and 3 show convincingly that BD reduces channel collapse. The overall results are modest improvements, but the point the authors make is that reducing collapse is correlated with improved performance.

I selected to reject the paper because in its current form a general readership will struggle to understand it. An expert familiar with the details of the field (not just the general area) can probably disentangle the text, but otherwise it's too convoluted.

An example note on math:
It would be great to reduce confusion and improve the consistency of math in the paper. In some parts of the manuscript, like section 3 "Notations", the symbols are well defined and clear to understand. In other locations, like section 1, an undefined \gamma is mentioned in the text, without definition. Is that \gamma the same as in section 3? What is its definition in section 1? 

Assuming that \gamma is as (loosely) defined in section 3, the reader can go back to section 1. This is an example of the convolution I mention above. For someone who has read the paper (and already read section 3), the \gamma in section 1 makes sense. 
review_title: Official Blind Review #3

Meta-review: The paper proposed Channel Equilibrium (CE) to overcome the over-sparsity problem in CNNs using 'BN+ReLU'. Experiments on ImageNet and COCO show its effectiveness by introducing little computational complexity. However the reviewers pointed a number of problems in the writing and the clarity of the paper. Although the authors addressed all the se concerns in details and agreed to make revisions in the paper, it's better for the authors to submit the revised version to another opportunity.
Decision: Reject