Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors introduce in this submission a locally-free weight sharing strategy for selecting effective network width. The rationale and intuition behind are well-grounded. Experiments on various datasets and pruning setups prove the validity. 

Strength:
+ The approach is well motivated and makes sense. The problem studied here is also important and could be of interest to a large audience.
+ Experiments are sufficient. The results are promising and well support the claim.
+ FLOPs-sensitivity bin considers factors including feature size and kernel size and seems to be independent of the total channel number, which, without douts, brings values.

Weakness:
- The proposed approach seems to be a  compromise between completely free weight and fixed weight, right? As a result, it would be good if the authors could elaborate the relation between the two.
- By utilizing the methods, my understanding is that the search space scales from O(N) all the way up to O(C_{2r+1}^{N}), no? This is a considerable amount of time required as compared to the single network width. Please provide some discussion along this line.
- The influence of the super network should be detailed. Intuitively, higher degrees of freedom will lead to better results. The authors should provide more analysis along this line.
- The writing can be enhanced. Please go over the manuscript and make sure all the grammar errors have been taken care of. 









review_title: Interesting Attempt on Network Width Search

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper explores the weight sharing schema in one-shot width search and proposes a locally free weight sharing strategy (CafeNet). By splitting each width candidate into base channels and free channels, CafeNet makes a compromise between fixed weight pattern and full freedom pattern. Such strategy can reduce the search complexity and improve the performance ranking, w.r.t. different width, in the supernet. Experiments on various tasks, including classification, detection and attribute recognition, are well provided to support the effectiveness of the proposed method. The final results are quite promising.

Strengths:
1) The paper is well written and easy to follow. The motivation is clearly explained by an example and the problem formulation.
2) The idea of locally free weight sharing is interesting. Such a solution for the previous fixed weight sharing seems sound.
3) Experiments with additional analyses are well provided.

I have the following concerns and suggestions:
1) Missing some relevant papers. OFA[1] and TF-NAS[2] introduce width search by dynamically choosing the channels. The authors should cite and explain the differences.
2) Is there any correlation between the degree in Eq. (4) and the searched accuracy under a fixed FLOPs?
3) The bin size of Eq. (9) makes me confusing. As shown in experiments, β can be less than 1 and the second term in the right side of Eq. (9) is also less than 1. Thus, the bin size bi (i.e., number of channels in a bin) is less than 1 channel. Please explain it in detail.
4) Why lambda=0 achieves the best accuracy in Fig. 2(b)? It is in conflict with the statement “As shown in Fig. 2(b), our 0.5-FLOPs MobileNetV2 on CIFAR-10 improves 0.92% accuracy from lambda=0 to lambda=1”.
5) I suggest the authors to split 1G group in Table 1, as done in Table 11.
6) In Algorithm1, both the supernet and the total number of epoch are defined as N.

Although some details make me a little confusing, the experimental analyses and the intuitive solutions of locally free weight sharing in one-shot width search are quite informative and helpful to the NAS community. I suggests the authors to release their code and would like to see the authors’ responses. 


[1] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, Song Han. Once-for-All: Train One Network and Specialize it for Efficient Deployment. ICLR, 2020.

[2] Yibo Hu, Xiang Wu, Ran He. TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search. ECCV, 2020.

review_title: An interesting idea with convincing results

final_score: rating: 6: Marginally above acceptance threshold
review_content: Most existing methods follow a manually ﬁxed weight sharing pattern, leading to the difficulty that estimates the performance of networks with different widths. To address this issue, this paper proposes a locally free weight sharing strategy (CafeNet) to share weights more freely. Moreover, this paper further proposes FLOPs-sensitive bins to reduces the size of the search space. Specifically, this paper divides channels into several groups/bins that have the same FLOPs-sensitivity and searches for promising architectures based on the divided groups. Extensive experiments on several benchmark datasets demonstrate superiority over the considered methods. However, some important details regarding the proposed method are missing. My detailed comments are as follows.

Positive points:
1. Compared with the manually ﬁxed weight sharing pattern, this paper proposes a locally free weight sharing strategy (CafeNet), which allows more freedom in the channel assignment of a sub-network.

2. To reduce the size of the search space, this paper proposes to divide channels into several groups/bins (also called minimum searching unit) that have the same FLOPs-sensitivity.

3. The experimental results on image classification and object detection tasks show that the proposed method outperforms the existing methods by a large margin.

Negative points:
1. When training the super network, why the authors optimize the sub-network with the smallest training loss? More explanations are required.

2. Why the sensitivity of a layer should be calculated as Eqn. (7)? It would be better to provide more details about that.

3. Given a FLOPs constraint in Eq. (2), how to select a suitable width for each layer? Please discuss more and make it clearer.

4. Is it possible to find a sub-network with zero width ($c=0$) for a layer? If so, how to deal with this case when evaluating the sub-network?

5. The experimental results are inconsistent with the descriptions. In Figure 2(b), the performance of the proposed method goes worse with the decreasing of the $\lambda$. However, the authors state that “MobileNetV2 on CIFAR-10 improves 0.92% accuracy from $\lambda$ =0 to $\lambda$=1”.

6. The experimental comparisons in Table 1 are unfair. Compared with other methods (e.g. AutoSlim), the proposed method trains the models on ImageNet for more epochs (100 v.s. 300). More experiments under the same settings are required.

Minor issues:
1. In appendix A.13, “… and the bin evolving speed α in Section 3.4” should be “… and the bin evolving speed α in Section 3.3”.
review_title: Official Blind Review #2

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this paper, the authors introduce a new weight sharing pattern to search for the width in a network layer. Besides, FLOPs-sensitive bins is proposed to measure the real FLOPs of a single channel at a layer and further reduce the search space. The paper proposes a locally free weight sharing mechanism where the channels in a layer are split into base channels and free channels. Compared with conventional fixed weight sharing pattern where the leftmost channels are assigned as the sub-network, the proposed locally free pattern increases more flexibility while the search space also scales at O(n). The proposed FLOPs-sensitive bins forces the layers with larger FLOPs sensitivity to have fewer channels, thus reducing the search space at a fixed FLOPs. Experimental results on several datasets show that the proposed CafeNet outperforms many other width search algorithms. The searched network experimentally achieves high performance with tiny FLOPs budgets.

What I like about this paper in that: 
1.	The motivation and intuition are reasonable, which is to design a more flexible weight sharing pattern for network width search. 
2.	Experiments are sufficient, thorough and carefully designed. Experimental results can support the objective of proposed methods. The searched network achieves remarkable performance with tiny FLOPs budgets.
3.	The paper is well written and organized. The work is easy to follow and be reproduced.
4.	The proposed methods have high generality and might be used on any convolutional network.

Some minor concerns or suggestion about this paper:
1.	The searching and training algorithms (max-max selection and min-min optimization) should be described in more detail.
2.	The free channels are the neighborhood of the c-th channel in this paper, but I think more channels on the right should be included in the zone.
review_title: An interesting weight sharing mechanism for network width search

Meta-review: The paper proposed locally free weight sharing strategy (CafeNet) for searching optimal network width. The proposal is a nice tradeoff between manually fixed weight sharing pattern (too small search space) and completely free weight sharing pattern (too large search space). The *originality* and *significance* are clearly above the bar. The paper is related to the general interests of deep learning research and its *applicability* deserves a spotlight presentation.

It seems the *clarity* can still be improved, so please carefully revise the paper following the reviews. BTW, I am very curious, why "locally free weight sharing strategy" goes to a short name CafeNet? I went over the paper but I didn't find the answer. Perhaps the name of the proposal should also be explained...
Decision: Accept (Spotlight)