Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper extends on the existing network slimming approach, computing a heterogenous width for each layer. The layer width is computed by solving a multi objective optimization problem based on pareto distribution.

Pros:

The paper is well motivated and well written.
The experiments are run across mutliple datasets and models. And also compared with existing approaches in the literature.
Cons:

The main aspect lacking in the paper is the novelty. In the three steps of the approach explained in "Pareco" section in Algorithm 1, the first and third step already exist in Network slimming approaches. And Step 2 - solving for 'c' is a multi objective optimization which directly exists in BoTorch library. So, I find lack of novelty and do not learn much or take away much reading this paper.

Also, by additionally solving for 'c', I do not realize what is the value add in terms of performance. In resnet-like architectures, there is no significant improvement in accuracy- as shown in Fig. 2 (a) to (l). The results in Mobilenet is slightly better as compared to NS - ~1 to 1.5% improvement, which is not significant, in my opinion. This is further established by interpreting, Fig 3(c) - for all different values of n (unless extremely high), we achieve a similar performance by the proposed approach.

Thus, I believe this paper is basically a hyperparameter tune over existing NS approaches, results in marginally not-so-sure significant improvement in results.
review_title: Lack of Novelty

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new multi-objective optimization method for slimmable neural networks and jointly optimizes network architecture and weights. However, the use of multi-objective optimization methods for networks is not new. The advantage of this paper over existing work in this regard is not clear. Besides, the key term “Pareto-aware” in the title is not clearly defined. 

review_title: PareCO: Pareto-aware Channel Optimization for Slimmable Neural Networks

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper directly extends the “Slimmable network” by using per-layer width multipliers to allow more flexible network configurations. PareCO mainly aims to optimize both width multipliers and shared weights while considering Pareto-optimal. (Accuracy & Speed) Because the problem is intractable, the authors adopt the Bayesian optimization model of CE loss and FLOPs. To improve search speed, several methods (temporal similarity, binary search) are also introduced. Compared to previous work (SLIM), PareCO consistently achieves better Pareto-optimal performance.

The paper is clearly written and has some significance. Related works are well established. Additional techniques, such as storing query for BO, binary searching, restrict to 1000 configurations, makes the proposed algorithm practical. The experimental result seems quite promising.

In my opinion, the novelty of the paper is not heterogenous width multiplier nor supernet-like training. Many works state that per-layer channel sparsity is important, and the idea of per-layer width multiplier is not new. I think this paper has some novelty, especially the “Pareto-Aware” part, but is not significant.

Minor questions/suggestions:

1.	Several works in NAS target Pareto-Optimal architecture. Some use Bayesian optimization.  Consider adding them to the related works section... [Jin-Dong Dong, 2018] DPP-Net, [Jin-Dong Dong, 2018] PPP-Net, [An-Chieh Cheng, 2018] Searching Toward Pareto-Optimal Device-Aware Neural Architectures, [Md Shahriar Iqbal, 2020] FlexiBO, etc.

2.	Isn’t the CE re-evaluation part for every query in history too costly? Especially for a big dataset like ImageNet. Is this the reason for “20% extra overhead compared to Slim”?

review_title: Good work, successfully extends previous work with proper techniques

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a multi-objective optimization approach to jointly train both channel configurations and shared weights of slimmable networks. It is theoretically proven that by minimizing the cross entropy loss for the Pareto-optimal channel configurations, the joint optimization can be approximately accomplished. Based on the objective, this paper proposes several approximation algorithms and sampling strategies to obtain a target network with better trade-off curve.

Experiments are thoroughly conducted on a range of network settings and datasets. The results experimentally show that the channel optimization can improve the performance and efficiency of slimmable networks. This paper is logically organized and the motivation is based on sufficient theoretical supports.

A few constructive criticisms or concerns as follows:
1.	The approximation algorithms and theorem 3.2 in this paper points out several ways to approximate the joint-learning objective. However, the authors did not provide descriptions about the intuition or advantages of the proposed approximation methods. Detailed theoretical analysis on the discrepancy between approximation and the ideal objective had better to be provided.
2.	Experiments cannot directly support the authors’ conclusion “less over-parameterized networks benefit more from joint channel and weight optimization” because the experiments cannot eliminate the impact of other variables.
3.	It might be difficult to reproduce this research and the experiment results since the search space is huge and the authors did not report the search cost (e.g. GPU-days and memory cost).

review_title: A joint optimization approach with rigorous theoritical support; concerns about evidence and reproducing

Meta-review: The paper extends the work of “slimmable networks” in that it aims to find a single set of weights suitable for multiple FLOP/accuracy tradeoff (or memory/accuracy tradeoff). The main novelty of the paper is in adapting known techniques from bayesian optimization (BO) to the setting at hand, resulting in a modified training technique. The experiments show a performance lift when compared against the original slimmable networks, as well as other approaches called “two stage” that alternate between optimizing the weights and the architecture.
The paper provides a practical approach to an important problem yielding non-trivial results. The main weakness of the paper seems to be its novelty. Although it is not possible to naively apply the multi-objective optimization with NAS techniques, the reviews seem to indicate that the innovation required to do so is not sufficient to meet the ICLR bar. This is indeed a borderline case, but given the competing papers, my tendency is towards rejecting the paper.

Decision: Reject